Author Archives: paulborile

About paulborile

I’m a multi-skilled IT professional with a good all-round supervisory and technical expertise. Extensive, 20+ years of professional experience in software development allowed me to investigate computer science and software engineering inside out. During these years I built up a solid base of design patterns, software architectures and programming languages such as C/C++, Golang, Java, Python, SQL, Assembly (and many others). I worked on mission-critical and multi-channel applications, applying distributed computing, messaging, image/data processing and computer graphics techniques. I faced both architecture design and systems rearchitecting, microservices introduction and technology migration as well as company wide adoption of new technologies/methodologies multiple times. As an entrepreneur I have built and grown teams and development organizations from the ground up (internal/out sourced/at customer site) focusing on software engineering methodologies as well as recruiting, budget/financial control and operations support. I am particularly interested in software testing methodologies, software quality metrics and tools to make software development faster and better. Currently leading the Italian development team for ScientiaMobile Inc, a Reston (US) based startup focused on image optimizing CDN and mobile detection technologies and services. Born in Dearborn Michigan and living in Italy since many years now I speak fluently both English and Italian, studied French and learned some Russian while working for some time for a Olivetti/Aeroflot project.

glibc 2.25 bug : strstr() runs 10 times slower than on 2.24

Linux is used on 54.9% of the world websites : almost every application running on a linux machine uses the glibc which provides the core libraries to access almost every feature of a linux system. The Mighty Glibc started back in 1988 and is a wonderful and glorious project.
As far as the string functions are concerned the sse / avx optimized versions of these functions (strlen, strcpy, strstr, strcmp and more) are up to 10 times faster than their corresponding standard c implementations (which for example you might find in the libmusl) when run on a sse/avx capable cpu.

We rely a lot on glibc string functions and that’s why we found that glibc 2.25 introduced some optimization on the AVX capable processors and this disabled sse* optimizations for methods that don’t have a avx2 optimized implementation (strstr, strcat, and I’m afraid parts of the math functions). For further details go here.
The bug affects ubuntu 18, debian 10, fedora 26 to 28.
A fix will come for sure, hopefully in glibc 2.29.

Update on November 3, 2020 : This bug was fixed in the package glibc – 2.27-3ubuntu1.3

Measuring memory footprint of a linux/macosx application

If you’re selling an API or an application which is deployed on production systems, one of the questions your customers might ask you is what is the memory footprint of your API/application in order for them to account for an increase of memory requirements due to using your product. After some research I think that the best tool for measuring and debugging any increases/decrease of your mem footprint is valgrind –tool=massif together with ms_print reporting tools.

Massif is a Heap memory profiler and will measure how much/when you allocate heap memory in your code and show the involved code. Run :

valgrind --tool=massif

this will execute the code and generate a massif.out.<pid> file that you may visualize with

ms_print massif.out.<pid>

Take a ride, the output is absolutely useful and you will have an histogram of how much memory is used at every sampling moment.

Bounds checking, leak checks and race conditions check with gcc sanitize

If you are coding since many years in C or C++ you probably hit many tools that helped in finding memory leaks in your code as well as did bounds checking to avoid buffers overrun. Those were PurifyPlus (rational), BoundsChecker (numega). Then valgrind cameout and all linux developers were happy to run valgrind on their code to find leaks and bounds overruns.

Since gcc 4.8 and llvm 3.1 Sanitize has been introduce in the compiler as a compile option which gives you the ability to check for :

leak and bounds overrun : -fsanitize=address
race conditions : -fsanitize=thread

Using sanitizer in your normal Test environment (we use c++ google test framework to test api code) will drastically change the quality of your software products (C, C++ or GO) by allowing you to spot most of the bounds overrun and leaks during the testing phase, both in your code and in the code that you include not written by you.

In my sample makefiles we build all component that we use in test in debug mode (-g3) and with sanitize settable from outside via an environment variable :

SANITIZE = -fsanitize=address

CXXFLAGS = -std=c++11 -pthread -fpermissive -isystem ../../googletest/include -isystem ../../googletest -DDEBUG -g3 $(SANITIZE)

Compiling a library with sanitize is :

$ make -f libfh.mk debugclean debuglinux
rm -rf fh.do fh.mo libfh-debug.a
cc -DDEBUG -g3 -fsanitize=address -O0 -fPIC --std=c99 -Wall -Wextra -Wcomment -pthread -I . -c fh.c -o fh.do
ar crs libfh-debug.a fh.do

Compiling tests :

$ make -f libfh-test.mk testclean testlinux
rm -rf TestMain.o TestThread.o TestFH.o CommandLineParams.o testfh *.gcda *.gcno gtest-all.o ../../*_test_results.xml
g++ -std=c++11 -pthread -fpermissive -isystem ../../googletest/include -isystem ../../googletest -DDEBUG -g3 -fsanitize=address -I.. -I ../../libutil/src -c -o TestMain.o TestMain.cpp
g++ -std=c++11 -pthread -fpermissive -isystem ../../googletest/include -isystem ../../googletest -DDEBUG -g3 -fsanitize=address -I.. -I ../../libutil/src -c -o TestThread.o TestThread.cpp
g++ -std=c++11 -pthread -fpermissive -isystem ../../googletest/include -isystem ../../googletest -DDEBUG -g3 -fsanitize=address -I.. -I ../../libutil/src -c -o TestFH.o TestFH.cpp
g++ -std=c++11 -pthread -fpermissive -isystem ../../googletest/include -isystem ../../googletest -DDEBUG -g3 -fsanitize=address -I.. -I ../../libutil/src -c -o CommandLineParams.o CommandLineParams.cpp
g++ -isystem ../../googletest/include -I ../../googletest/ -pthread -c ../../googletest/src/gtest-all.cc
g++ -o testfh -fsanitize=address TestMain.o TestThread.o TestFH.o CommandLineParams.o gtest-all.o -L ../ -lfh-debug -lpthread

Now let’s run a test (after having introduced a bug) :


paul@paul-bigdesk:~/git/clibs/libfh/test$ ./testfh --gtest_filter=*hash_with_struct*
Running main() in file c-libraries/libfh/test/TestMain.cpp
Now running test executable: ./testfh
Note: Google Test filter = *hash_with_struct*
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from FH
[ RUN ] FH.hash_with_struct
=================================================================
==23907==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60200000eb5f at pc 0x7f37a13bc964 bp 0x7fff66eca0b0 sp 0x7fff66ec9858
WRITE of size 16 at 0x60200000eb5f thread T0
#0 0x7f37a13bc963 in __asan_memcpy (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x8c963)
#1 0x466298 in fh_insert /home/paul/git/clibs/libfh/fh.c:390
#2 0x414ed5 in FH_hash_with_struct_Test::TestBody() /home/paul/git/clibs/libfh/test/TestFH.cpp:486
#3 0x45683e in void testing::internal::HandleSehExceptionsInMethodIfSupported&lt;testing::Test, void&gt;(testing::Test*, void (testing::Test::*)(), char const*) (/home/paul/git/clibs/libfh/test/testfh+0x45683e)
#4 0x44fdf6 in void testing::internal::HandleExceptionsInMethodIfSupported&lt;testing::Test, void&gt;(testing::Test*, void (testing::Test::*)(), char const*) (/home/paul/git/clibs/libfh/test/testfh+0x44fdf6)
#5 0x433737 in testing::Test::Run() (/home/paul/git/clibs/libfh/test/testfh+0x433737)
#6 0x4340cf in testing::TestInfo::Run() (/home/paul/git/clibs/libfh/test/testfh+0x4340cf)
#7 0x4347c2 in testing::TestCase::Run() (/home/paul/git/clibs/libfh/test/testfh+0x4347c2)
#8 0x43b8b5 in testing::internal::UnitTestImpl::RunAllTests() (/home/paul/git/clibs/libfh/test/testfh+0x43b8b5)
#9 0x457e16 in bool testing::internal::HandleSehExceptionsInMethodIfSupported&lt;testing::internal::UnitTestImpl, bool&gt;(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (/home/paul/git/clibs/libfh/test/testfh+0x457e16)
#10 0x450c6e in bool testing::internal::HandleExceptionsInMethodIfSupported&lt;testing::internal::UnitTestImpl, bool&gt;(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (/home/paul/git/clibs/libfh/test/testfh+0x450c6e)
#11 0x43a35b in testing::UnitTest::Run() (/home/paul/git/clibs/libfh/test/testfh+0x43a35b)
#12 0x406819 in main /home/paul/git/clibs/libfh/test/TestMain.cpp:38
#13 0x7f37a07c582f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
#14 0x406538 in _start (/home/paul/git/clibs/libfh/test/testfh+0x406538)

0x60200000eb5f is located 0 bytes to the right of 15-byte region [0x60200000eb50,0x60200000eb5f)
allocated by thread T0 here:
#0 0x7f37a13c8662 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98662)
#1 0x466222 in fh_insert /home/paul/git/clibs/libfh/fh.c:384
#2 0x414ed5 in FH_hash_with_struct_Test::TestBody() /home/paul/git/clibs/libfh/test/TestFH.cpp:486
#3 0x45683e in void testing::internal::HandleSehExceptionsInMethodIfSupported&lt;testing::Test, void&gt;(testing::Test*, void (testing::Test::*)(), char const*) (/home/paul/git/clibs/libfh/test/testfh+0x45683e)
#4 0x44fdf6 in void testing::internal::HandleExceptionsInMethodIfSupported&lt;testing::Test, void&gt;(testing::Test*, void (testing::Test::*)(), char const*) (/home/paul/git/clibs/libfh/test/testfh+0x44fdf6)
#5 0x433737 in testing::Test::Run() (/home/paul/git/clibs/libfh/test/testfh+0x433737)
#6 0x4340cf in testing::TestInfo::Run() (/home/paul/git/clibs/libfh/test/testfh+0x4340cf)
#7 0x4347c2 in testing::TestCase::Run() (/home/paul/git/clibs/libfh/test/testfh+0x4347c2)
#8 0x43b8b5 in testing::internal::UnitTestImpl::RunAllTests() (/home/paul/git/clibs/libfh/test/testfh+0x43b8b5)
#9 0x457e16 in bool testing::internal::HandleSehExceptionsInMethodIfSupported&lt;testing::internal::UnitTestImpl, bool&gt;(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (/home/paul/git/clibs/libfh/test/testfh+0x457e16)
#10 0x450c6e in bool testing::internal::HandleExceptionsInMethodIfSupported&lt;testing::internal::UnitTestImpl, bool&gt;(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (/home/paul/git/clibs/libfh/test/testfh+0x450c6e)
#11 0x43a35b in testing::UnitTest::Run() (/home/paul/git/clibs/libfh/test/testfh+0x43a35b)
#12 0x406819 in main /home/paul/git/clibs/libfh/test/TestMain.cpp:38
#13 0x7f37a07c582f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

SUMMARY: AddressSanitizer: heap-buffer-overflow ??:0 __asan_memcpy
Shadow bytes around the buggy address:
0x0c047fff9d10: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c047fff9d20: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c047fff9d30: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c047fff9d40: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c047fff9d50: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=&gt;0x0c047fff9d60: fa fa fa fa fa fa fa fa fa fa 00[07]fa fa 05 fa
0x0c047fff9d70: fa fa 00 00 fa fa 00 00 fa fa 00 fa fa fa 00 fa
0x0c047fff9d80: fa fa 00 fa fa fa 00 fa fa fa 00 fa fa fa 00 fa
0x0c047fff9d90: fa fa 00 fa fa fa 00 fa fa fa 00 fa fa fa 00 fa
0x0c047fff9da0: fa fa 00 fa fa fa 00 fa fa fa 00 fa fa fa 00 fa
0x0c047fff9db0: fa fa 00 fa fa fa 00 fa fa fa 00 fa fa fa 00 fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Heap right redzone: fb
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack partial redzone: f4
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
==23907==ABORTING

You easily get to were the bug is fh.c line 390; we decremented the allocated size by 1:

new_opaque_obj = malloc(fh->h_datalen-1);

Easily you can switch to race conditions checking my using

$ make SANITIZE=-fsanitize=thread -f libfh-test.mk testclean testlinux

Design for simplicity : software options

xkcd.com

(I was highly inspired by Evan Martin post before trying to recap in this post)
Think twice before adding an option to a software; options are :

bad user experience : good software just works without asking to many things to the user.
a maintenance cost : an option is a feature you have to maintain/test for the lifetime of the software
documentation cost : every option, even the most cryptical, should be documented to the user, explained and you should provide some examples of use. When you normally try to do this you end up removing the option from documentation
normally options are a way to delegate to user a decision that the designer failed to take : “shall we open on the last opened document or shall put an options ‘always open on blank document’ ?
remember the paradox of choice
options setting generates bad GUIs : they are hard to present, group or organize.

On the other hand consider that any change you make will probably break someone’s workflow, no doubt, so the temptation to create an option to enable the previous operational mode is high.

The old windows search wizard, awesome, read this : https://www.joelonsoftware.com/2000/04/12/choices/

WTF/Minute : code quality measure unit

Some day your code will be reviewed, modified or even just read to understand the underlying algorithm by someone else that is not you that wrote the code….

My point here is that readability/maintainability of code is by far the biggest quality of a piece of software. Being able to read the code and understand it rapidly makes everything easier, either bugfixing or modifying/enhancing the code. Think that the next one that might be reading the code might not be as experienced as you with that language so

avoid using unnecessarily complicated constructs that feed you coder ego but slow down the maintenance activity. You are a good coder, no doubt, but if your code is not readable by the average coder that will have to then you are not making a good service to your colleagues
Make clear what your code is doing and, if necessary use a comment to clarify what is not evident from the code (I’m not a huge fan of comments because most of them don’t tell anything more than the code). Use comments to refer to a requirement/user-story that is being implemented.
Stick to the language style guidelines that were set for your team/company
Tests are code too and often are read/reviewed more than real code : don’t fuck up in tests code just because “this is just a test, this code will not run in production”

Discussion on code quality is huge of course and these are simple rules that will impact on maintainability of your code (since a reviewer is doing the same activity of the next coder which will have to work on your code : read it). If you want to dig deeper into code quality measurement I’ll suggest this work from Idan Amit and Dror G. Feitelson really interesting : https://arxiv.org/pdf/2007.10912.pdf

Comic comes from here

Paul Stephen Borile

software teams management, golang/c coding, bass player, biking addicted