A few words about Solange project

For near­ly nine months, I am work­ing on a pro­ject called Solange which con­sists of De­vOps plat­form which makes it pain­less to de­ploy new pro­jects in pro­duc­tion. Sur­pris­ing­ly, I haven't writ­ten too much about it, not be­cause it's a se­cret (the source is pub­licly avail­able), but most­ly be­cause it is dif­fi­cult to get start­ed and de­scribe the pro­ject which is not yet in its fi­nal shape.

I think now is a good time to de­scribe what was done, and what are the ex­pec­ta­tions for the fu­ture.

Cre­at­ing vir­tu­al ma­chines

You know Chef and Pup­pet, right? Well, one part of my pro­ject is very sim­i­lar to those two prod­ucts. It al­lows to de­scribe a vir­tu­al ma­chine or a set of vir­tu­al ma­chines, and then de­ploy them across the data cen­ter. With­out en­ter­ing in the de­tails about Chef and Pup­pet, I will in­stead de­scribe how my own one is or­ga­nized.

Pro­files and in­stances

The pro­ject uses pro­files to de­scribe sim­i­lar ma­chines and in­stances to de­scribe the de­tails of a spe­cif­ic ma­chine.

For in­stance, every vir­tu­al ma­chine which hosts Mon­goDB with repli­ca sets shares the same pro­file. This blog re­lies on two Mon­goDB ma­chines work­ing in pair, while the redi­rec­tion web­site uses a dif­fer­ent pair of Mon­goDB ma­chines; Iris ser­vice uses an­oth­er pair, etc. They all use the same pro­file, but also six dif­fer­ent in­stances, for six VMs.

This mod­el makes it easy to de­fine the frame, such as “Post­greSQL with load bal­anc­ing” or “In­ter­nal SVN serv­er with mir­ror­ing” or “Pub­lic read-only SVN serv­er”, and then set the specifics in­side the in­stances, for ex­am­ple “Back­up di­rec­to­ry for Post­greSQL data­base” or “The num­ber of mir­rors for an in­ter­nal SVN serv­er”.

Both pro­files and in­stances use JSON as its pri­ma­ry way to con­fig­ure ba­sic el­e­ments, such as the list of apt pack­ages to in­stall, and bash or Python scripts (or any ex­e­cutable) as a way to con­fig­ure things which are not sup­port­ed through JSON con­fig­u­ra­tion.

This al­lows to change ba­sic set­tings (such as the NFS back­up path) with ease by chang­ing the JSON file, while hav­ing the pow­er to do any­thing we need in bash.

Cur­rent state

The pro­ject is ac­tive­ly used in­ter­nal­ly, and the whole in­fra­struc­ture re­lies on it. This blog is host­ed on vir­tu­al ma­chines de­ployed with a sin­gle com­mand. The same ap­plies to our source servers. Ac­tu­al­ly, every web­site and web ser­vice host­ed at Pel­i­can De­sign & De­vel­op­ment uses the pro­ject, in­clud­ing the PXE ma­chines which were used to de­ploy Lin­ux to the serv­er which hosts the vir­tu­al ma­chines.

I don't con­sid­er the pro­ject to be ready for pub­lic: while it is us­able, it ter­ri­bly lacks doc­u­men­ta­tion and tests. The fact that the pro­ject grew or­gan­i­cal­ly while I was learn­ing the ba­sics of Lin­ux, Python and Node.js led to the dif­fi­cul­ty to test and to doc­u­ment the pro­ject cor­rect­ly.

Still, the pro­ject is us­able, so you're free to play with it if you want.

Fur­ther de­vel­op­ment

In the next months, the goal is to make it pos­si­ble to make this plat­form less man­u­al. In fact, while au­toma­tion is here, con­fig­u­ra­tion should still be changed by hand, and it is still re­quired to type the com­mand which de­ploys or de­stroys a vir­tu­al ma­chine.

What I need, on the oth­er hand, is to be able to tell to the sys­tem that I need, for in­stance, two Mon­goDB repli­ca sets in pair and two small in­stances of ap­pli­ca­tion servers for a giv­en pro­ject. I don't care what should be the con­fig­u­ra­tion: what I care is that I have a giv­en URI point­ing to the pro­ject with­in the SVN repos­i­to­ry, and I want it to run in pro­duc­tion. What IP ad­dress­es should the sys­tem chose or what ports should be opened is not my prob­lem: it is at the stage when I de­fine the orig­i­nal pro­files.

This rep­re­sents a huge amount of work, but will also make it ex­treme­ly easy to de­ploy pro­jects. Ide­al­ly, this can even al­low to sell com­put­ing pow­er to oth­er com­pa­nies. For in­stance, what if a small com­pa­ny has a choice: ei­ther pro­gram­mers may spend days or weeks un­der­stand­ing Ama­zon AWS or Win­dows Azure, or they can have all the in­fra­struc­ture they need in one click at Pel­i­can De­sign & De­vel­op­ment, and start de­vel­op­ing the pro­ject right now?

Us­ing con­tin­u­ous de­ploy­ment

While the pre­vi­ous­ly de­scribed part of the pro­ject is es­sen­tial, the goal is also to have a full con­tin­u­ous de­ploy­ment plat­form. What I mean by that is that go­ing to a web­site, click­ing on a bunch of but­tons and see­ing the pro­ject de­ploy it­self in the wild is nice, but not enough. With the num­ber of re­vi­sions an av­er­age pro­ject can have per day, it is sim­ply out of ques­tion to do those things by hand.

There­fore, de­ploy­ment should be au­to­mat­ed even fur­ther. It's not a ques­tion of telling to the sys­tem that I need two failover app servers and two repli­ca sets for a giv­en pro­ject in a SVN repos­i­to­ry, but rather telling: “Here is my pro­ject. Every time I do a com­mit, test the new re­vi­sion and in­tel­li­gent­ly de­ploy it in pro­duc­tion, en­sur­ing that there will be no down­time and no in­for­ma­tion loss.”

Get­ting to this lev­el of au­toma­tion would be even hard­er. Nev­er­the­less, I know this is doable, and I'm pret­ty sure I will get there, one fea­ture at a time. The road will not be easy, but I know my goal and I know what I should do for the next few days; and that's all I need.