Geo-spatial in-memory caching using Rtree

Adam Lev-Libfeld

A long distance runner, a software architect, an HPC nerd (order may change).

Latest posts by Adam Lev-Libfeld (see all)

Having to save a lot of geo-spatial data is a task rarely one passes without being scared for one’s life. Whether you are using a DB like postGIS, an in-memory data store like geoRedis or some other tool, handling geo-spatial data usually mean using RTree. Using RTree directly can ave some very positive effects on your system performance, so we thought we’ll let you in on this industry secret. Continue reading “Geo-spatial in-memory caching using Rtree”

Geo-spatial in-memory caching using Rtree

Logging Geo-Spatial-Data to ELK from Python

Adam Lev-Libfeld

A long distance runner, a software architect, an HPC nerd (order may change).

Latest posts by Adam Lev-Libfeld (see all)

Kibana, on paper, offers a beautiful and easy way to inspect your geo data on a dashboard widget out of the box. This is all very well until you try to use that feature from python, only to find out that the Logstash lib automatically maps all of these geo-point type fields to string. Mapping in ES is no child’s play but even solving mapping issues* didn’t completely made the problem go away.

Continue reading “Logging Geo-Spatial-Data to ELK from Python”

Logging Geo-Spatial-Data to ELK from Python

Zopfli Optimization: Free Bandwidth (?)

Adam Lev-Libfeld

A long distance runner, a software architect, an HPC nerd (order may change).

Latest posts by Adam Lev-Libfeld (see all)

Source: Zopfli Optimization: Literally Free Bandwidth

This is a little off subject, but space (= bandwidth) optimization is just as important as algorithmic optimization. That said – there is a cost to applying compression solutions globally in your system, as this article gently notes. Handle with care.

Link

Must Not Fail

Adam Lev-Libfeld

A long distance runner, a software architect, an HPC nerd (order may change).

Latest posts by Adam Lev-Libfeld (see all)

No system is fault-proof, but high volume, high velocity and high throughput (or whichever flavor of HPC you practice toady) systems often are expected to be. Every time we state otherwise we either lie to our clients, our colleagues, or worse – ourselves.

Continue reading “Must Not Fail”

Must Not Fail

Fast Learner Tip: A Visual Introduction to Machine Learning

Adam Lev-Libfeld

A long distance runner, a software architect, an HPC nerd (order may change).

Latest posts by Adam Lev-Libfeld (see all)

What is machine learning? See how it works with our animated data visualization.

Source: A Visual Introduction to Machine Learning

 

When approaching a performance problem this method of train and test is, sadly, not yet fully applicable. This is why we have to keep our software as scalable as possible, to enable it to fit on the go with increasing (or falling) demand.

The ability to train on real time data (or learn from mistakes) using a closed feedback loop is a common method to do adaptive machine learning, and is exactly the same as we use to determine how many instances of our software currently needs to be up.

In Storm, circles in topologies are usually looked upon as a perversion, but with a couple of simple tricks you can always move the model out of Storm to a datastore (Redis maybe?), and separate the training bolt from the predicting (or classifying, or testing, depending on what stage you are) bolt.

Link

Breaking the monolith – How to design your system for both flexibility and scale – Part 4: The Message

Adam Lev-Libfeld

A long distance runner, a software architect, an HPC nerd (order may change).

Latest posts by Adam Lev-Libfeld (see all)

This post is part of a series. you’d probably like to read it from the beginning. Or check out the previous post in the series

Those of us who had the chance to experience the exquisite pleasure of getting a formal education in the field of software engineering were told a fabulous tale. This tale of extreme bravery and audacity  included mythical creatures and ideas – like a zero-downtime zero-loss infrastructure, instant globally synchronizing data (allowing the creation of what is sometimes referred to as the “single point of truth”),  and of course – the object oriented system.

Continue reading “Breaking the monolith – How to design your system for both flexibility and scale – Part 4: The Message”

Breaking the monolith – How to design your system for both flexibility and scale – Part 4: The Message

Breaking the monolith – How to design your system for both flexibility and scale – Part 3: The embarrassingly parallel truth

Adam Lev-Libfeld

A long distance runner, a software architect, an HPC nerd (order may change).

Latest posts by Adam Lev-Libfeld (see all)

This post is part of a series. you’d probably like to read it from the beginning. Or check out the previous post in the series

 

We are about to dig a hole. The hole will be 8 meter deep, 1m long and 1m wide. We know, from experience as a professional hole digger it will take us about 8 hours to dig all the way down. But this time there’s a difference – we will get some help.

together we stand

So our boss decided to get us some help, 7 people worth of help to be exact. But of course in life as in life, and nothing is free – as we are now eight, she wants us to dig the same hole in one hour. Trying to explain why this is not possible, you plead, you beg, you even say trying to dig that particular hole in an hour is like asking nine women to create a human baby in one month. Alas, the boss is the BOSS, and she will not budge – she wants that earth moved.

Continue reading “Breaking the monolith – How to design your system for both flexibility and scale – Part 3: The embarrassingly parallel truth”

Breaking the monolith – How to design your system for both flexibility and scale – Part 3: The embarrassingly parallel truth