- Airbnb: Ruby on Rails, MySQL, Amazon Web Services (AWS)
- Uber: Go, Node.js, Java, Python, PostgreSQL, MySQL, Redis, Kafka
- GitHub: Ruby on Rails, Go, Erlang, MySQL, Redis
- WhatsApp: Erlang, FreeBSD, Yaws, Ejabberd, Mnesia DB
- LinkedIn: Java, Kafka, Hadoop, Cassandra, Oracle, Voldemort (LinkedIn’s own distributed database)
- Netflix: Go, Gluster, Apache Tomcat, Hive, Chukwa, Cassandra, Hadoop, MySQL, Amazon Web Services (AWS)
- Slack: PHP, Java, MySQL, MongoDB, Kafka
- Facebook: PHP (HHVM), React, GraphQL, MySQL, Cassandra
- Twitter: Scala, Java, Ruby, MySQL, Aurora, Redis
- Spotify: Python, Java, ZeroMQ, PostgreSQL, Cassandra, Google Cloud Platform (GCP)
- Pinterest: Python (Django), Java, Redis, Cassandra, Kafka
- Snapchat: Python, Java, PHP, Cassandra, Google Cloud Platform (GCP)
- Dropbox: Python, Go, Rust, MySQL, Amazon Web Services (AWS)
- Zoom: C++, Java, MySQL, Kafka, Amazon Web Services (AWS)
- Shopify: Ruby on Rails, MySQL, Redis, Liquid (templating), Google Cloud Platform (GCP)
- TikTok (ByteDance): Python, C++, Java, Nginx, MySQL, Kafka
- Reddit: Python (Django), PostgreSQL, Redis, RabbitMQ, Amazon Web Services (AWS)
- Instagram: Python (Django), React, PostgreSQL, Redis, Amazon Web Services (AWS)
Author Archives: paulborile
The charm of complication
(or the Attraction for Complexity) There is a very common tendency in computer science and it is to complicate solutions. This complication is often referred as incidental/accidental complexity i.e. anything we coders/designers do to make more complex a simple matter. Some times this is called over engineering and stems from the best intentions :
- Attraction to Complexity: there’s often a misconception that more complex solutions are inherently better or more sophisticated. This can lead to choosing complicated approaches over simpler, more effective ones.
- Technological Enthusiasm: developers might be eager to try out new technologies, patterns, or architectures. While innovation is important, using new tech for its own sake can lead to unnecessary complexity.
- Anticipating Future Needs: developers may try to build solutions that are overly flexible to accommodate potential future requirements. This often leads to complex designs that are not needed for the current scope of the project.
- Lack of Experience or Misjudgment: less experienced developers might not yet have the insight to choose the simplest effective solution, while even seasoned developers can sometimes overestimate what’s necessary for a project.
- Avoiding Refactoring: In an attempt to avoid refactoring in the future, developers might add layers of abstraction or additional features they think might be needed later, resulting in over-engineered solutions.
- Miscommunication or Lack of Clear Requirements: without clear requirements or effective communication within a team, developers might make assumptions about what’s needed, leading to solutions that are more complex than necessary.
- Premature Optimization: trying to optimize every aspect of a solution from the beginning can lead to complexity. The adage “premature optimization is the root of all evil” highlights the pitfalls of optimizing before it’s clear that performance is an issue.
- Unclear Problem Definition: not fully understanding the problem that needs to be solved can result in solutions that are more complicated than needed. A clear problem definition is essential for a simple and effective solution.
- Personal Preference or Style: sometimes, the preference for certain coding styles, architectures, or patterns can lead to more complex solutions, even if simpler alternatives would suffice.
- Fear of Under-Engineering: there can be a fear of delivering a solution that appears under-engineered or too simplistic, leading to adding unnecessary features or layers of abstraction.
RIP Niklaus Wirth
We all know Pascal language (less know about Algol or Oberon OS) and we might have heard that
“software is getting slower more rapidly than hardware is becoming faster”
or that
“What Intel giveth, Microsoft taketh away”
I prefer this quote :
“Reducing complexity and size must be the goal in every step—in system specification, design, and in detailed programming. A programmer’s competence should be judged by the ability to find simple solutions, certainly not by productivity measured in number of lines ejected per day. Prolific programmers contribute
to certain disaster.”
You find them all here “A plea for lean software“. if you make code please read this 1995 paper.
2023 Reading list (updated as we go)
Interesting https://medium.com/@jsoverson/was-rust-worth-it-f43d171fb1b3
Reforming Unix : https://github.com/Ericson2314/baccumulation/blob/main/reforming-unix.adoc
Managing big C language codebases can be challenging but is definitely possible : this very interesting post talks about curl https://daniel.haxx.se/blog/2023/12/13/making-it-harder-to-do-wrong/
Agile has failed ? I don’t think so : https://medium.com/developer-rants/agile-has-failed-officially-8136b0522c49 Anything applied as a religion is doomed to fail and the same is for Agile. You can’t take any methodology and apply it “as is” to a company or project or dev-team; you need to adapt it and not make the company/team adapt to it.
What I hope is that we don’t trow away the good ideas of Agile (iterations/continuous delivery, attention to technical excellence, simplicity (avoiding over engineering) just to mention a few).
State of the developer ecosystem : https://www.jetbrains.com/lp/devecosystem-2023/
Margareth Hamilton, the first software engineer story : https://levelup.gitconnected.com/the-untold-story-of-the-errorless-code-written-by-a-woman-that-took-man-to-the-moon
Avoid Try/Catch https://betterprogramming.pub/try-catch-considered-harmful and remember what google recommends about exceptions (and c++ in general)
Thanks Mafe : Maniac by Benajmin Labatut, see the Alpha-Go documentary
Agile roadmaps : Now, Next, Later approach https://developingskills.substack.com/p/now-next-later
Reasons why you burnout swe : https://engineercodex.substack.com/p/how-to-burnout-a-software-engineer My list in order of priority : 1. don’t ship code (there is nothing worse than working months on something that is not going to prod or is lingering in the deploy queue) 2. Not trusting your engineers by telling them in fine detail how to do things. 3. Lack of Recognition (not reward, about the difference).
So google uses design documents to describe design of new components : https://www.industrialempathy.com/posts/design-docs-at-google/
From a friend of mine substack, Chris Hedges talking about war :
“But these words give me a balm to my grief, a momentary solace, a little understanding, as I stumble forward into the void.”
Readability (and thus Maintainability) is the most important attribute of a code base and Google knows it : https://engineercodex.substack.com/p/how-google-writes-clean-maintainable
An open source 3D universe simulator with support for more than a billion objects : https://zah.uni-heidelberg.de/gaia/outreach/gaiasky/ based on data from Gaia mission
Resources on Microservices https://blog.quastor.org/p/scaling-microservices-doordash-6c35
How Diffusion Models work i.e. how does an AI generate images : https://developer.nvidia.com/blog/improving-diffusion-models-as-an-alternative-to-gans-part-1/
Watch out for Thundering herds
Engineering Metrics : https://hybridhacker.email/p/diving-into-engineering-metrics
My next read : http://lawsofsimplicity.com/ John Maeda
System design and the cost of architectural complexity : https://dspace.mit.edu/handle/1721.1/79551
Simplicity was the key for Instagram backend (in 2011) : https://engineercodex.substack.com/p/how-instagram-scaled-to-14-million
Simplicity mindset : https://betterprogramming.pub/3-tips-to-adopt-a-simplicity-mindset-when-designing-software-711b95328062
Fastly supports go on Edge : https://www.fastly.com/blog/announcing-standard-go-support-for-fastly-compute
Go WASI support : https://go.dev/blog/wasi
Any company providing support for a product would want to have something like this https://www.fixie.ai/
Meta Launches AI Code Writing tool https://www.theverge.com/2023/8/24/23843487/meta-llama-code-generation-generative-ai-llm : you can test it here : https://labs.perplexity.ai/
Whats App : https://newsletter.systemdesign.one/p/whatsapp-engineering
Photographs that are like films https://www.rencontres-arles.com/en/expositions/view/1447/gregory-crewdson
I find it so true : https://intenseminimalism.com/2013/conways-law/
Jerome Lanier on music and what Reality is in the end : https://www.newyorker.com/culture/the-weekend-essay/what-my-musical-instruments-have-taught-me
We lost Kevin (The Condor) Mitnick ( https://www.dignitymemorial.com/obituaries/las-vegas-nv/kevin-mitnick-11371668 ) : TCP is a much more secure protocol also thanks to him.
Yoshua Bengio (CA) and John Bunzl (UK), moderated by Nico A. Heller on strengths and limitations of current artificial intelligence, why it may become a dangerous instrument of disinformation, why superintelligent AI may be closer (years) than most previously expected (decades) and how this could yield to catastrophic outcomes – from AI-driven wars to the extreme risk of extinction. https://www.youtube.com/watch?v=07c1ZRUQOeY. Notes, general concepts from Bengio talk :
AI currently perceive the world and make sense of it with images, sound and text, generating content in all 3 areas.
Current systems are not at the level of human intelligence, they master what psychologist call system 1 intelligence (intuition) : react to any question/context, no reasoning (or little reasoning) – arithmetic : simple operations with numbers ok, more complex (of the type we will need paper and pencil) they make mistakes.
System 2 intelligence : explicit reasoning, you can plan, imagine. Ex:
driving left hand roads after having driven right hand all life. We take 1 hour to adapt because we don’t use intuition but reasoning. AI will get there : how much time will take to bridge the gap between s1 and s2 ? nobody knows, could be close or take 10 years.
Training data currently needs to be filtered to remove data that appears to be insulting, homophobic or racists, dangerous, inadequate. AI will get there too to avoid the need of preparing training material.
Machines that are as intelligent as we are will be inevitably more intelligent than us because they are machines : immortal, don’t sleep, can exchange info at high speeds like if they were a huge brain. Humans have culture to try to simulate this.
- pure intelectual power, completely detached from the goal. The goal will make the difference between a “good” AI and an “evil” one.
- humans cannot turn off compassion (or just most people can’t) as it has been hardwired into us by evolution; machines can
Stating a goal in a precise way is probably impossible so even AI with good goals might behave evil.
Goals are not expressable, we can only give partial specifications
We’ll get to the point where the game will be : who’s AI is bigger/better/faster ?
Most important thing to reduce the probability of bad behavior connected to AI is to reduce the actors, materials, information, proliferation (like we did for atomic bombs)
Regulatory frameworks are necessary.
Go : inside map implementation https://www.youtube.com/watch?v=Tl7mi9QmLns and https://phati-sawant.medium.com/internals-of-map-in-golang-33db6e25b3f8
Go 10 years after : https://blog.carlmjohnson.net/post/2023/ten-years-of-go-good-bad-meh/ and why we can live without subclassing https://hynek.me/articles/python-subclassing-redux/
Extreme readability https://www.moderndescartes.com/essays/readability/
AI can’t create software, only code : https://stackoverflow.blog/2023/06/26/the-hardest-part-of-building-software-is-not-coding-its-requirements
Some WASM Stuff :
– cloudflare wasm runtime https://github.com/cloudflare/workerd
– wasmtime runtime : https://github.com/bytecodealliance/wasmtime
– wasi-libc : still no thread support (june 2023) https://github.com/WebAssembly/wasi-libc
Ping Pong Programming : TDD + Pair Programming : https://www.agileconnection.com/article/ping-pong-programming-enhance-your-tdd-and-pair-programming-practices
Using AI in Ping Pong programming : https://www.mechanical-orchard.com/post/can-ai-play-the-tdd-pairing-game
AI Technology map : https://a16z.com/2023/06/20/emerging-architectures-for-llm-applications/
You build it, you own it : https://blog.alexewerlof.com/p/you-build-it-you-own-it
AI generated sort algorithms make it into LLVM/C++ https://www.nature.com/articles/s41586-023-06004-9
Superfast hash https://github.com/wangyi-fudan/wyhash
Google leaked internal document on AI : https://www.semianalysis.com/p/google-we-have-no-moat-and-neitherkkjhnmmmm
Compression algorithms benchmarks : https://github.com/inikep/lzbench
What’s inside training data for LLM https://arxiv.org/pdf/2101.00027.pdf
Cache Optimization Models and Algorithms : https://arxiv.org/pdf/1912.12339.pdf
ChatGPT Is a Blurry JPEG of the Web https://twitter.com/paulborile/status/1647640363555758083
Ordinals https://docs.ordinals.com/overview.html
NO Pause on AI development : https://medium.com/enrique-dans/a-pause-on-the-development-of-ai-its-not-going-to-happen-d4f894816e82
Cheatsheet for golang test, benchmarks, profiling https://github.com/samonzeweb/profilinggo
Yet another browser https://mullvad.net/it/browser promising to minimize tracking and fingerprinting.
everything cli https://www.commandlinefu.com/commands/browse
Amazon Is Making a New Web Browser ? https://gizmodo.com/amazon-prime-new-web-browser-survey-1850224922?utm_source=tldrnewsletter
Interview with OpenAI’s Greg Brockman: GPT-4 isn’t perfect, but neither are you https://techcrunch.com/2023/03/15/interview-with-openais-greg-brockman-gpt-4-isnt-perfect-but-neither-are-you/?utm_source=tldrnewsletter&guccounter=1
On jpeg-XL https://cloudfour.com/thinks/on-container-queries-responsive-images-and-jpeg-xl/#jpeg-xl-the-holy-grail-image-format
https://cloudinary.com/blog/the-case-for-jpeg-xl
On crypto and thieves :
“In February 2022, a hacker stole 120,000 wrapped Ethereum from Wormhole, a cross-blockchain bridge” https://newsletter.mollywhite.net/p/oasis-defi-centralization – subscribe to Molly White newsletter for unbiased crypto news.
B Corporation Certification : https://en.wikipedia.org/wiki/B_Corporation_(certification) “As of September 2022, there are 5,697 certified B Corporations across 158 industries in 85 countries.” https://www.bcorporation.net/en-us/
https://www.positive.news/economics/five-trends-sustainable-business/
https://lumalabs.ai/ : acquire 3d assets anywhere with a phone
https://roaringbitmap.org/ compressed bitmaps
Code RED in google : https://medium.com/@ignacio.de.gregorio.noblejas/can-chatgpt-kill-google-6d59742ee635
Node.js is aging like milk https://javascript.plainenglish.io/node-js-is-aging-like-milk-now-whats-next-a4e726ae668f :
Ryan Dahl, creator of Node.js moved to Go.
TJ Holowaychuk, creator of Express.js framework moved to Go.
Boston dynamics showing off https://youtu.be/-e1_QhJ1EhQ
Chat GPT4 talk https://www.theverge.com/23560328/openai-gpt-4-rumor-release-date-sam-altman-interview
Fyne IDE, currently work in progress https://github.com/fyne-io/defyne
1995 paper by the Niklaus Wirth “A Plea for Lean Software” https://cr.yp.to/bib/1995/wirth.pdf
2022 Reading list
Pingora : Cloudflare new rust based proxy for edge servers https://blog.cloudflare.com/how-we-built-pingora-the-proxy-that-connects-cloudflare-to-the-internet/
How much memory your video card has : https://www.cyberciti.biz/faq/howto-find-linux-vga-video-card-ram/
Rewriting a monolith into microservices (or just into a more maintainable monolith) : the Strangler pattern https://www.redhat.com/architect/pros-and-cons-strangler-architecture-pattern
Context switching costs in sw development : https://www.infoq.com/articles/multitasking-problems/
Wasmedge runtime https://wasmedge.org/book/en/index.html
Python compiler that produces native machine code (10/100x speed gain) https://github.com/exaloop/codon
How AVIF compares to other image formats https://storage.googleapis.com/avif-comparison/index.html
Some tools for enumerating : https://github.com/OJ/gobuster, https://github.com/OWASP/Amass
Updates on the twitter affair : https://newsletter.pragmaticengineer.com/p/the-scoop-twitters-ongoing-cruel
Reverse engineering Swift physics
Pelikan cache, a C/Rust memcached type cache server made in twitter http://www.pelikan.io/
Giza 3D http://giza.fas.harvard.edu/giza3d/
Memory allocator used in Go language : https://google.github.io/tcmalloc/design.html
Slab Allocation : https://hammertux.github.io/slab-allocator
Linux virtual FS filename lookup : https://www.kernel.org/doc/html/latest/filesystems/path-lookup.html
Linux dentry cache and slabtop command
Your wordpress site can feed mastodon https://fedi.tips/the-fediverse-beyond-mastodon/#WordPress
Image Format wars :
– Google removing JXL support from chrome : https://chromium-review.googlesource.com/c/chromium/src/+/3988999
– reasons https://www.phoronix.com/news/Chrome-Dropping-JPEG-XL-Reasons
– comments : https://cloudinary.com/blog/the-case-for-jpeg-xl
– support for JXL in android : https://issuetracker.google.com/issues/259900694?pli=1
Bare metal or cloud services : https://levelup.gitconnected.com/how-we-reduced-our-annual-server-costs-by-80-from-1m-to-200k-by-moving-away-from-aws-2b98cbd21b46
New Directions in Cloud Programming : https://www.cidrdb.org/cidr2021/papers/cidr2021_paper16.pdf
I know you all know but, just a reminder :
package main
import "fmt"
func main() {
var m = 0.1
var n = 0.2
fmt.Println(m + n)
var a = 0.1
var b = 6.0
fmt.Println(a * b)
var o = 0.6
var p = 0.3
fmt.Println(o + p)
var f = 1.39
fmt.Println(f * f)
const c = 1.39
fmt.Println(c * c)
}
0.30000000000000004
0.6000000000000001
0.8999999999999999
1.9320999999999997
1.9321
Reducing Cognitive load (Golab2022 notes) in code :
- Line of Sight : 1st level, second level for error conditions
- eliminate else, return early, avoid extra nesting, wrap in function
- align the happy path
- remove misterious booleans
- Package Names : concise and clear, move on the calling side. No util/common/ packages, split
- Pure functions as much as possible
- pass with & to indicate we change the object
- Don’t lie to the reader
Interesting WASM talk from https://xeiaso.net/
Organize you software assets https://stackoverflow.blog/2022/09/19/i-spent-two-years-trying-to-do-what-backstage-does-for-free/ spotify open sourced their internal tool
VCF East 2019 Bryan Kernigham interviews Ken Thompson : how unix, pipes, and other things like grep came out just to run nroff/troff for Bell Labs patents office https://youtu.be/EY6q5dv_B-o
Engineering ladders https://github.com/jorgef/engineeringladders
Stackoverflow engineering https://hanselminutes.com/847/engineering-stack-overflow-with-roberta-arcoverde
Stackexchange 1.3 bil pageview/month : https://stackexchange.com/performance
Airbnb path from monolith to microservices https://medium.com/qe-unit/airbnbs-microservices-architecture-journey-to-quality-engineering-d5a490e6ba4f
Perche’ l’innovazione e’ cosi’ difficile in italia : qui
Questo non vale solo per la digitalizzazione, vale per l’industrializzazione e per il capitalismo. Prendo in prestito la domanda (e la risposta) di Roberto Maragliano “Perché i temi della maturità sono ancora così? Ce lo dice Camillo Olivetti (padre di Adriano) in un appunto del 1927: “L’istruzione della nostra borghesia ha un fondamento prettamente anti-industriale. Noi siamo ancora i figli dei Latini che lasciarono ai servi e ai liberti i lavori industriali e che in ben poco conto li ritennero, tanto che ci tramandarono i nomi dei più mediocri proconsoli e dei poetucoli ed istrioni che dilettarono la decadenza romana, ma non ci ricordarono neppure i nomi di quei sommi ingegneri che costruirono le strade gli acquedotti e i grandi monumenti dell’Impero Romano” (da Paolo Bricco, Adriano Olivetti, un italiano del Novecento, Rizzoli, appena uscito)”.
End to end tracing : Jaeger (opentelemetry)
Consistent Hashing used for load balancing resources :
- envoy : ring hash and maglev
- haproxy
- gRPC client load balancer
Porting python2 code to python3
Interesting point of view on “complete rewrite” vs “enhance/evolve the existing” from dropbox tech blog
T Shape Engineers
Perkeep .. “Throughout our life, we all continue to generate content …”
Peer to peer hypermedia protocol : https://ipfs.io/
Building and Testing in-kernel (Rust code to be discussed at next Linux Plumbers : https://lpc.events/event/16/page/193-proposed-microconferences and discussed here : https://arstechnica.com/gadgets/2021/03/linus-torvalds-weighs-in-on-rust-language-in-the-linux-kernel/
Linux Foundation Real Time OS : https://www.zephyrproject.org/learn-about/
Microservice patterns : https://microservices.io/patterns/index.html
Broken Window Theory applied to software ( initial statement in “The Pragmatic Programmer” by Andrew Hunt and David Thomas, 1999 if I’m not wrong) interesting review and interesting to meet another fan of simplicity and minimalism in code.
Again on avoiding accidental complexity (I wrote something on this in the past) https://searchsoftwarequality.techtarget.com/tip/How-to-prevent-accidental-complexity-in-software-development
Interesting :
Super, 10X, rockstar — these are names for developers who produce a lot of code or work in many development areas. Such developers sometimes build a reputation — and an ego. “These super developers usually make their code unnecessarily complex just because they can,” Rials said. Other team members struggle to comprehend and edit the complex code. These colleagues might even compliment the super developer about the code’s complexity, which encourages more of it. Businesses should empower junior developers to take a fresh look at code.
Bill Rials, associate director and professor at Tulane University’s School of Professional Advancement
How common the pattern above is ..
Composer, music theorist, architect, performance director and engineer https://en.wikipedia.org/wiki/Iannis_Xenakis (I love multi potential people). Known for his studies on Music generation software : author of UPIC, which inspired many other tools like Iannix, HighC, UPISketch. I was able to run HighC …
Tz0, a video game made to listen music
The first ecologist ? https://www.rachelcarson.org/
Clearness when she explains, clearness of vision, clearness of thinking : Mariana Mazzucato
https://projecteuler.net/ take some time to solve math things
The importance of Rubber Ducking : a funny tool to do it alone ..
Ethical resources : sustainable, mindful, transparent, open and much more. Here’s a list of tools (browsers, search engines, email services, web hosting, tools for team collaboration, messaging, and a lot more) for a more ethical internet.