TwitterFlickrYouTube

Archives de catégorie: G33k

Oui, il ne faut pas cacher son moi primaire…. Ça marche pas à la longue.

Android : what if « git push » becomes your Play Store publish command

Posté le par Posté dans G33k, Programmes .

Ever wondered why you need these repetitive tasks each time (in these three pictures below) ?
apk from intellij
intellij2

playstore

Not only you run the risk of losing time, but you can go wrong because of a lack of concentration (by publishing in prod instead of alpha), or you can even forget to publish your app intermediary updates. Your users might lose patience while waiting for some fixes, or some features.

Now think about how devops teams works. The team developers change a piece of code, than ensure nothing is broken, then let the software factory prepare, package, deploy and monitor the program.

How to achieve this, without money nor time ?
Start by using :

  • a source control management : either a private one, or BitBucket, or GitHub.
  • an Application Lifecycle Management tool : maven or gradle
  • a Continuous Integration service : sure there is Jenkins, but you can use a free cloud CI : http://travis-ci.org
  • an account of the Play Store of course.

This example below will use GitHub + maven + Travis :

  • go to your play store params > API Access > Create a service account. Display this item in the Google Developers Console and download the .p12 file
  • make sure you know where your .keystore file is to sign your APK. If you haven’t got one yet, please visit the android developers help https://developer.android.com/studio/publish/app-signing.html
  • add your repository to travis and configure a .travis.yml file in the root folder of your repository
  • install the travis client with gem install travis
  • base64 encode both the keystore and the p12 file, then concat them together (cat key.p12.b64 account.keystore.b64 > credentials.b64). Do not place the file in your repo, instead move it to the parent folder.
  • ask travis to add it securely without comprimising the security (travis encrypt-file ../credentials.b64 --add). You can add the credentials.b64.enc securely to your repository without fear.
  • have a look at this travis build file : https://github.com/libetl/singIn/blob/alpha/.travis.yml. We have the openssl command that was asked to be added, but the ALM must be aware of both keystore and keypass passwords.
  • There is a tip to unpack a single .enc file into both the keystore and the .p12 file (.travis.yml lines 19-20). Split the file on the == token to know where the p12 file ends and where the keystore begins. base64 decode them both, and you are set…
  • on the maven command line, we ask to proceed to the tests (org.jacoco:jacoco-maven-plugin:prepare-agent test org.jacoco:jacoco-maven-plugin:report org.eluder.coveralls:coveralls-maven-plugin:report), then the packaging phase (android:manifest-merger install android:zipalign) then the publishing phase (android:publish-apk), and we pass as parameters the keystore and keypass secrets right from the travis hidden vars settings
  • now we just need to setup the android-maven-plugin so it is aware of how it must be configured.
    • The android manifest versionCode must be incremented for each publish, therefore we rely on the $TRAVIS_BUILD_NUMBER value for the counter.
    • We ask the plugin to override the manifest from the repository (by specifying the output file name of the manifest-merger target).
    • We must fill in the service account e-mail and the path to the .p12 file
    • We must provide a path to the keystore, with the following secrets given as properties : keystorepass, keypass
    • Have a look at this pom.xml file to know more about the challenge : https://github.com/libetl/singIn/blob/alpha/pom.xml
  • publish a first version of your app if it is not already done (by the classic workflow with your IDE or bash commands)
  • change any file on your git repository, commit the changes, push them…
  • go to your travis app job, and see it working : https://travis-ci.org/libetl/singIn/builds/171238769

Wait about 10-15 minutes, and your app is deployed on the play store automatically :
playstore2

From now on, each time you change something in your codebase, will be published on the Play Store… painless.
Now you can say : to deploy on the Play Store, just do git push

Let your code know… your code

Posté le par Posté dans G33k, Programmes .

Has this ever happened to you ?

… you work on a legacy project, and there is this method called `getAccount` which sounds simple.
But this method is a compound one.
1°) It actually calls a database A to fetch a user Id.
2°) Then completes the data with a B Web service based on LDAP
3°) Then completes it with some extra data based on the account preferences from a service C
4°) It updates the database with the found data (yes, « getAccount » writes something… in a legacy project, you will certainly see that)
5°) It returns all the data.

What if you call getAccount from another point of view ? If you are the caller of the method, you could even do this :

if (getAccount (user) != null) {
String userName = getAccount (user).getName ();
}

Ugh… Did you realize that these 4 lines will call different external services 3 * 2 = 6 times… It has awful performance, and can be harmful (= with side effects) if the method getAccount is not idempotent (which is likely to happen in a legacy project)

What you need is not (really) refactoring. The technical debt is too tough to be attacked right now. What you need IS knowledge about the code. How can you know what is happening in the code ? Think painless, think about code to understand code …

I have created what I call a DiveDeepAnalyzer. The aim of this is to understand which sensitive methods are called directly or indirectly in each method.

https://gist.github.com/libetl/00bc6079b3dd91af55bb6cf8229e942a

Write down the list of sensitive calls (webservices, LDAP, databases, cache, drivers, hardware), and the algorithm will annotate each and every method that is likely to use these calls.

How to ?
1°) Copy this class in your project,
2°) change the params in the main method (you will need to exclude the DiveDeepAnalyzer class itself from the scan)
3°) run the class.

This analyzer uses spoon, an OSS made by a research lab in France (inria). You can find more about spoon at this address : http://spoon.gforge.inria.fr/

My Git hint : a feature per commit

Posté le par Posté dans G33k, Programmes .

Have you had a look on your project git log ? It is often an enumeration of changes which are sometimes not explained correctly… « bugfixes », « some changes », « work in progress », « pull request review ».

wip git log

The log what you would want to see is a log of all features recently added from the JIRA (or any other issue tracker) board.
Moreover, having your features grouped in one commit will be a way to know all of the necessary changes for each issue (because a git diff will reveal all the changes of a feature at once)

Here is how it can be done :

– As a setup, create this command alias in your .gitconfig file (I assume that your main branch is called « master » here.)

[alias]
singleCommit = "!f() { git pull origin master --no-edit && git push && git reset --soft origin/master && git commit -m \"$1\" && git push -f; }; f"

– create a git branch from « master »
– commit everything you want inside this branch, don’t hesitate to commit or push any correction for a mistake you found.
– create a pull request if you need to, review the comments, commit / push, and so on.
– when everything is ready and the code is ready to be merged, find the JIRA code (or your issue tracker issue id) (we will name it MYPROJECT-XXXX)
– issue this command : git singleCommit "MYPROJECT-XXXX a comment for this feature" (change this text to the aim, or the title of the feature)
– if the command fails, your branch is not synced with master. Resolve the conflicts, git add . and git commit, then go back to the instruction before.
– now you can merge. There will be ONE commit for all the content of the branch.

Jackson : Make the most of POJO versus Immutable Objects

Posté le par Posté dans G33k, Programmes .

There is a simple dilemma that comes into my thoughts :
-> I have an object, right… class A { String b;int c}
-> It must be filled automatically thanks to unmarshalling from a json string {"b":"val1","c":2}
-> I want it to be immutable because I don’t want to alter my input, or to trust some insecure hashcode/equals methods.

Therefore : I must have a deserializable class, with final fields, a comprehensive constructor, and a builder pattern.

That sounds awkward with jackson. But is it possible ? Yes, it is, and I’ll prove it.
https://gist.github.com/libetl/853029faf7999c98159f36d1c229c961#file-a-java

Neat, but let’s complicate this attempt a bit. Suppose we have a polymorphic object to be deserialized. It is a paradigmatic issue between Java and Json : Java is typed, Json does not care.
let’s declare this in pseudo DSL : class TripSegment { TransportMode mode; SegmentDetails detailsOfTheSegment }
with enum TransportMode { FLIGHT, TRAIN, BOAT, CAR_SHARING, LOCAL_TRANSIT }
and with several details classes like this one : class LocalTransitDetails implements SegmentDetails {String lineCode;DateTime timeOfDeparture;DateTime timeOfArrival;String departureStationName;String arrivalStationName;}

We want neither to create a TripSegmentBean as a POJO, nor to have a non jackson deserializer… So how to ?
Here is how I could successfully answer to that problem :
https://gist.github.com/libetl/853029faf7999c98159f36d1c229c961#file-tripsegment-java

Now, I can handle my unmarshalled value object as if I built it manually. Now forget about dozer or BeanUtils. Let’s concentrate on the workflow.

curl4j

Posté le par Posté dans G33k, Programmes .

Would you enjoy being able to cURL something within one command line in java ?

Seems sometimes annoying to instantiate an HttpPost, or even an HttpClientBuilder, with its fluent interface.

What I suggest you is to use my code sample, to enhance it and to transform it in your customized cURL command. You can even add custom arguments to it, so the commands will be shorter to type.

How to use it ? Like a normal curl !
Response curlResponse = curl(" -X'POST' -H'Content-Type:application/x-www-form-urlencoded' -d 'token=mytoken&channel=mychannel' 'https://slack.com/api/chat.postMessage'")

Here you are, with a curl response that you are able to read easily with the HttpClient API.
Where is this code sample ? There : https://gist.github.com/libetl/edd72cca5464aa403395029430360344

EDIT : 2016-07-22 : https://github.com/libetl/curl and https://mvnrepository.com/artifact/org.toile-libre.libe/curl

BTW, this blog is now 8 years old, and this is the 200th post… way to go.

Convert List(a,b,c,d) into Map(a->b,c->d) in Java 8

Posté le par Posté dans G33k, Programmes .

Let input be a List of String, you want it to become Map<String, String> where even elements are the keys, and odd elements are the values. Works only when key elements are unique.

Map<String, String> mappedValues = input.stream().collect(Collectors.toMap( Function.identity(), input::indexOf)).entrySet().stream().collect(Collectors.toMap(
entry -> {return entry.getKey();}, entry -> {return entry.getValue() == input.size() − 1 ? −1 : entry.getValue() + 1;})).entrySet().stream().filter(entry -> {return entry.getValue() % 2 == 1;}).collect(Collectors.toMap(entry -> {return entry.getKey();}, entry -> {return input.get(entry.getValue());}));

My opinion on the « Best-so-far » architecture

Posté le par Posté dans Blabla, G33k .

This post is not to talk about something I have made, but to talk about something I have lived with.

First, what is it ?

« Best-so-far » (or BSF) means to work with the latest pieces of software each and every day.

It relies on several IT infrastructure components : a software factory, a source code management, a build scheduler and an integration monitoring.

The software factory

It is a software whose role is to collect all the libraries across a company and to release the binaries for the employees, partners and customers.
It can be implemented by a storage access on a platform, or a machine, or a shared drive.
It also contains archived binaries of the previous versions of each component (for redundancy purposes), but is mostly focused on providing the latest framework chunks, with the latest features and the latest bugfixes.
All of these binaries can be used dynamically to build and release other binaries, and so on…

The SCM (source code management)

It is a set of tools to optimize the branches code management in a team (allowing several users to commit on the same project with the least possible conflicts).
Most of the teams appreciate to use standard tools like git, svn, mercurial, cvs, ClearCase, perforce, bazaar. Some others wanted to create their own scm tool (thinking of adl, http://www.maruf.ca/files/caadoc/CAAWsmQuickRefs/).
It is often managed by having one single integration branch where every team member has to merge in after finishing a task.

The build scheduler

It has two modules : a batch whose role is to execute some actions on some files, and an UI to help the users to administrate the jobs to be run each day.
It uses the latest version from « the SCM » to build it and install the packages on « The software factory ». It also contains some details about the errors found while trying to build or release something.

The integration monitoring

It is a UI tool with red and green lights to inform all of the employees that a build failed or suceeded. The most common standard software existing are Jenkins, Bamboo, CircleCI, CruiseControl, or Drone. Some companies prefer to create their own, with better or worse usability (thinking of ReleaseWork). The integration monitoring collects data from « the SCM », « the build scheduler » and « The software factory » to gives an insight of what is working and what is not.

2. Why do we use it ?

The thinking behind this implementation is to wonder how different dev teams can work across a company on the same product without suffering from waiting for each other. The expectation is to « break the silos », and to get the teams in a rhythm.
Letting a dev team beneath another in terms of software dependencies will help the dependent team to update the subsequent code as soon as possible. For example, a team building a polygon extrusion lib will update every day the dependencies to benefit from the work of the maths library. As a consequence, each lib will get the « Best-so-far » state of each lib when it comes to releasing to the employees, partners, customers. And so on…

3. Is it good ? I don’t know. Is it easy ? NO, it is NOT.

This is where I will express opinions of that practice.
I have worked for a company using that practice since birth. The « break the silos » fantasy is far away from reality.
I have never worked in such a partitioned way of thinking company.
The thing that turned wrong is that the work streams who managed the frameworks were separated from those who managed the BSF architecture.
As a result, a lot of improvements in this architecture landed in a total contradiction of the dev teams practices :
– The team handling the integration monitoring is pressured to release the build. The team members try to identify who is late by seeing who has worked on « the SCM ». They identify some culprits, without any consideration about the team responsibility. This team is not appreciated because it is seen like a toilet lady to the developers. « Have you fixed your issue ? The next release must be submitted by noon. I won’t leave the office until it is fixed and working »
– The team working for « the SCM » delivers some features that are useful for some users. Not all. All the other teams are forced to work on ONE SCM that does not fit their needs (if the company uses cvs instead of git, no one is allowed to work with git, and the Internet proxy must forbid any request sent to github or bitbucket).
– « The software factory » uses a common repository which can work with any kind of project lifecycle tool. You must be able to package Java classes, C++ projects or DSL scripts at once and to send them on the same storage access. Do we know a PL tool which is working on anything ? Yes, but it is really primitive. « make » can do the job. That was the most awkard answer I could ever think of. To understand it, just imagine that I work on a Java project but I have to build and to release my software using the « make » C++ tool. And last that not least, just imagine that the choosen storage can be as clever as a Windows share (that is it, samba share) !
– « The build scheduler » is maybe the tool which is the least risky to maintain, because the responsibility lies on the dev teams. It basically consists in launching some tasks and display the logs. Here, no specific tool to an environment. Windows shells are launched, that is all. Neither any JUnit parsing, nor any performance report.

Last but not least,

what is the most disturbing item for developers in the BSF way ?

→ You are not going to believe it, but it is the principle itself. Working on snapshots itself is at once a good, and a bad practice. Imagine that your company makes yogurts. It wants to rely on a single milk provider, which is considered as the most talented milk provider locally. « We don’t need fallback providers, we use the best one, the best-so-far ». Now imagine that a brutal and unexpected disease cause the death of the whole livestock in one night. How the company is going to adapt to that situation ?
So does a company with a « Best-so-far » architecture. You cannot work with any other provider, and you cannot work with older but functional versions of the frameworks.
The activity stops in the hope that the situation gets better in the future. In my opinion, that explains why my previous company decided to release one product version per year at most.

Replayable REST logs on spring-mvc

Posté le par Posté dans G33k, Programmes .

« What happens on this app ? I cannot figure out what these logs mean… »
That is one of the typical sentences we are likely to say when it comes to debugging in blackbox mode (without access to the code itself).

Although, there is one thing we all know : In a situation, the app creates some pieces of data and then send them to another object (whether it is a person or a service).

In some recent web architectures, with spring, we are likely to send REST requests to other services. We can customize how the data is sent and also how we should trace it.

The idea I want to underline is that a kind of logs that help is a « replayable log », which means that someone looking at the code can try a request once more to understand a failing scenario, step by step, accurately

Then, why not curl logs ? A curl log syntax is really short, understandable and concise. It helps to understand how the data is being sent to another actor.

How to start ?
1. You can copy this gist into your app : https://gist.github.com/libetl/2d2060ce892987a9b7a0bdcb9b234a93
2. Add it inside your RestTemplate in your App Spring Configuration :

@Bean
public RestTemplate thirdPartyServiceRestTemplate () {
final RestTemplate restTemplate = new RestTemplate();
//...
restTemplate.setInterceptors(java.util.Collections.singletonList(new org.toilelibre.libe.logs.CurlLogHttpRequestInterceptor()));
//...

return restTemplate ;
}

AWS SWF (Simple Workflow Service) : I want a status !

Posté le par Posté dans G33k, Programmes .

Hello,

This post will include a block of code to help you to monitor your SWF executions in your own console (rather than the AWS console itself).

You can include this simple class in your own code (do not notice me, use it) and transform it to fit your monitoring needs.

This code will :
– find the registered domain matching the domain name you want
– get and store the open executions in a list
– append and store the closed execution in the same list

If you issue workflowsFollowUp.toString (), you will get hierarchical and detailed data about :
– each step (DecisionTask or ActivityTask)
– the list of activities which were run, in order
– the datetime values (when it was started)
– if there are, the exceptions unmarshalled as real exceptions…

Example of result in the console :

----
Workflow type DemoWorfklowDecider.launch/1.0
Workflow bd4852d2-862f-4d7b-a837-2aca62ea09f2 with runId 2209T4DgTTd6iB82nZdAPXeKQ8iEv6S1dailgO3x9Jq/U=
Workflow status : CLOSED ; result : COMPLETED

Step 1 : WorkflowExecutionStarted ; date : Wed Oct 14 15:10:43 CEST 2015 ; input : ["[Ljava.lang.Object;",[10,true]]
Step 2 : DecisionTaskScheduled ; date : Wed Oct 14 15:10:43 CEST 2015
Step 3 : DecisionTaskStarted ; date : Wed Oct 14 15:10:43 CEST 2015
Step 4 : DecisionTaskCompleted ; date : Wed Oct 14 15:10:44 CEST 2015
Step 5 : ActivityTaskScheduled ; date : Wed Oct 14 15:10:44 CEST 2015 ; activity : DemoWorkflowActivity.waitForThenFailOrSuccess ; input : ["[Ljava.lang.Object;",[10,true]]
Step 6 : ActivityTaskStarted ; date : Wed Oct 14 15:10:44 CEST 2015
Step 7 : ActivityTaskCompleted ; date : Wed Oct 14 15:10:54 CEST 2015 ; result : true
Step 8 : DecisionTaskScheduled ; date : Wed Oct 14 15:10:54 CEST 2015
Step 9 : DecisionTaskStarted ; date : Wed Oct 14 15:10:54 CEST 2015
Step 10 : DecisionTaskCompleted ; date : Wed Oct 14 15:10:55 CEST 2015
Step 11 : WorkflowExecutionCompleted ; date : Wed Oct 14 15:10:55 CEST 2015

Activities run :
- DemoWorkflowActivity.waitForThenFailOrSuccess
----
Workflow type DemoWorfklowDecider.launch/1.0
Workflow 338b088e-b8be-48c7-996e-84a493788347 with runId 22+BzfJcLaGosAT44+kZkyATWB8uXnx7lLE5/bCJOKmZ4=
Workflow status : CLOSED ; result : FAILED

Step 1 : WorkflowExecutionStarted ; date : Wed Oct 14 14:39:51 CEST 2015 ; input :
Step 2 : DecisionTaskScheduled ; date : Wed Oct 14 14:39:51 CEST 2015
Step 3 : DecisionTaskStarted ; date : Wed Oct 14 14:39:52 CEST 2015
Step 4 : DecisionTaskCompleted ; date : Wed Oct 14 14:39:53 CEST 2015
Step 5 : WorkflowExecutionFailed ; date : Wed Oct 14 14:39:53 CEST 2015 ; reason : com.fasterxml.jackson.databind.JsonMappingException: No content to map due to end-of-input
at [Source: java.io.StringReader@762c40ce; line: 1, column: 1] when mapping key "null"
com.amazonaws.services.simpleworkflow.flow.DataConverterException: null when mapping key "null"
at com.amazonaws.services.simpleworkflow.flow.JsonDataConverter.fromData(JsonDataConverter.java:102)
at com.amazonaws.services.simpleworkflow.flow.pojo.POJOWorkflowDefinition$1.doTry(POJOWorkflowDefinition.java:74)
at --- continuation ---.(:0)
at com.amazonaws.services.simpleworkflow.flow.pojo.POJOWorkflowDefinition.execute(POJOWorkflowDefinition.java:66)
at com.amazonaws.services.simpleworkflow.flow.worker.AsyncDecider$WorkflowExecuteAsyncScope.doAsync(AsyncDecider.java:70)
Caused by: com.fasterxml.jackson.databind.JsonMappingException: N/A
at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:164)
at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3036)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:2978)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2098)
at com.amazonaws.services.simpleworkflow.flow.JsonDataConverter.fromData(JsonDataConverter.java:96)
at com.amazonaws.services.simpleworkflow.flow.pojo.POJOWorkflowDefinition$1.doTry(POJOWorkflowDefinition.java:74)
at com.amazonaws.services.simpleworkflow.flow.core.TryCatchFinallyContext.run(TryCatchFinallyContext.java:163)
at com.amazonaws.services.simpleworkflow.flow.core.AsyncEventLoop.executeQueuedTask(AsyncEventLoop.java:44)
at com.amazonaws.services.simpleworkflow.flow.core.AsyncEventLoop.executeAllQueued(AsyncEventLoop.java:33)
at com.amazonaws.services.simpleworkflow.flow.core.AsyncScopeContext.eventLoop(AsyncScopeContext.java:151)
at com.amazonaws.services.simpleworkflow.flow.core.AsyncScope.eventLoop(AsyncScope.java:74)
at com.amazonaws.services.simpleworkflow.flow.worker.AsyncDecider.eventLoop(AsyncDecider.java:327)
at com.amazonaws.services.simpleworkflow.flow.worker.AsyncDecider.decide(AsyncDecider.java:497)
at com.amazonaws.services.simpleworkflow.flow.worker.AsyncDecisionTaskHandler.handleDecisionTask(AsyncDecisionTaskHandler.java:50)
at com.amazonaws.services.simpleworkflow.flow.worker.DecisionTaskPoller.pollAndProcessSingleTask(DecisionTaskPoller.java:201)
at com.amazonaws.services.simpleworkflow.flow.worker.GenericWorker$PollServiceTask.run(GenericWorker.java:94)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Please find this code in Gist : https://gist.github.com/libetl/b117b321cda6bf539986

I wish you a happy monitoring.

How to read an ogg file in Java without JNI

Posté le par Posté dans G33k, Programmes .
  1. Import jorbis as a dependenyc of your project
  2. Read the howto : http://www.jcraft.com/jorbis/tutorial/Tutorial.html
  3. Copy the source code. It will work, but will only be able to play a sound (not to convert it)
  4. Look at my JorbisCleanConverter. (https://raw.githubusercontent.com/libetl/soundtransform/master/src/main/java/org/toilelibre/libe/soundtransform/infrastructure/service/audioformat/converter/JorbisCleanConverter.java) and copy everything (you will need to change the catch clauses).
  5. You are now able to transform an ogginputstream into a wav outputstream using a ByteArrayInputStream class. To get the stream entirely, just do .toByteArray(), you can write that in a file and use it as a real wav file.