Simple Dynamic Strings

Notes about version 2: this is an updated version of SDS in an attempt to finally unify Redis, Disque, Hiredis, and the stand alone SDS versions. This version is NOT binary compatible* with SDS verison 1, but the API is 99% compatible so switching to the new lib should be trivial.

Note that this version of SDS may be a slower with certain workloads, but uses less memory compared to V1 since header size is dynamic and depends to the string to alloc.

Moreover it includes a few more API functions, notably sdscatfmt which is a faster version of sdscatprintf that can be used for the simpler cases in order to avoid the libc printf family functions performance penalty.

How SDS strings work

SDS is a string library for C designed to augment the limited libc string handling functionalities by adding heap allocated strings that are:

• Simpler to use.
• Binary safe.
• Computationally more efficient.
• But yet... Compatible with normal C string functions.

This is achieved using an alternative design in which instead of using a C structure to represent a string, we use a binary prefix that is stored before the actual pointer to the string that is returned by SDS to the user.

+--------+-------------------------------+-----------+
| Header | Binary safe C alike string... | Null term |
+--------+-------------------------------+-----------+
         |
         `-> Pointer returned to the user.

Because of meta data stored before the actual returned pointer as a prefix, and because of every SDS string implicitly adding a null term at the end of the string regardless of the actual content of the string, SDS strings work well together with C strings and the user is free to use them interchangeably with other std C string functions that access the string in read-only.

SDS was a C string I developed in the past for my everyday C programming needs, later it was moved into Redis where it is used extensively and where it was modified in order to be suitable for high performance operations. Now it was extracted from Redis and forked as a stand alone project.

Because of its many years life inside Redis, SDS provides both higher level functions for easy strings manipulation in C, but also a set of low level functions that make it possible to write high performance code without paying a penalty for using an higher level string library.

Advantages and disadvantages of SDS

Normally dynamic string libraries for C are implemented using a structure that defines the string. The structure has a pointer field that is managed by the string function, so it looks like this:

struct yourAverageStringLibrary {
    char *buf;
    size_t len;
    ... possibly more fields here ...
};

SDS strings as already mentioned don't follow this schema, and are instead a single allocation with a prefix that lives before the address actually returned for the string.

There are advantages and disadvantages with this approach over the traditional approach:

Disadvantage #1: many functions return the new string as value, since sometimes SDS requires to create a new string with more space, so the most SDS API calls look like this:

s = sdscat(s,"Some more data");

As you can see s is used as input for sdscat but is also set to the value returned by the SDS API call, since we are not sure if the call modified the SDS string we passed or allocated a new one. Not remembering to assign back the return value of sdscat or similar functions to the variable holding the SDS string will result in a bug.

Disadvantage #2: if an SDS string is shared in different places in your program you have to modify all the references when you modify the string. However most of the times when you need to share SDS strings it is much better to encapsulate them into structures with a reference count otherwise it is too easy to incur into memory leaks.

Advantage #1: you can pass SDS strings to functions designed for C functions without accessing a struct member or calling a function, like this:

printf("%s\n", sds_string);

In most other libraries this will be something like:

printf("%s\n", string->buf);

Or:

printf("%s\n", getStringPointer(string));

Advantage #2: accessing individual chars is straightforward. C is a low level language so this is an important operation in many programs. With SDS strings accessing individual chars is very natural:

printf("%c %c\n", s[0], s[1]);

With other libraries your best chance is to assign string->buf (or call the function to get the string pointer) to a char pointer and work with this. However since the other libraries may reallocate the buffer implicitly every time you call a function that may modify the string you have to get a reference to the buffer again.

Advantage #3: single allocation has better cache locality. Usually when you access a string created by a string library using a structure, you have two different allocations for the structure representing the string, and the actual buffer holding the string. Over the time the buffer is reallocated, and it is likely that it ends in a totally different part of memory compared to the structure itself. Since modern programs performances are often dominated by cache misses, SDS may perform better in many workloads.

SDS basics

The type of SDS strings is just the char pointer char *. However SDS defines an sds type as alias of char * in its header file: you should use the sds type in order to make sure you remember that a given variable in your program holds an SDS string and not a C string, however this is not mandatory.

This is the simplest SDS program you can write that does something:

sds mystring = sdsnew("Hello World!");
printf("%s\n", mystring);
sdsfree(mystring);

output> Hello World!

The above small program already shows a few important things about SDS:

• SDS strings are created, and heap allocated, via the sdsnew() function, or other similar functions that we'll see in a moment.
• SDS strings can be passed to printf() like any other C string.
• SDS strings require to be freed with sdsfree(), since they are heap allocated.

Creating SDS strings

sds sdsnewlen(const void *init, size_t initlen);
sds sdsnew(const char *init);
sds sdsempty(void);
sds sdsdup(const sds s);

There are many ways to create SDS strings:

• The sdsnew function creates an SDS string starting from a C null terminated string. We already saw how it works in the above example.

• The sdsnewlen function is similar to sdsnew but instead of creating the string assuming that the input string is null terminated, it gets an additional length parameter. This way you can create a string using binary data:

  char buf[3];
  sds mystring;
  
  buf[0] = 'A';
  buf[1] = 'B';
  buf[2] = 'C';
  mystring = sdsnewlen(buf,3);
  printf("%s of len %d\n", mystring, (int) sdslen(mystring));
  
  output> ABC of len 3
  

  Note: sdslen return value is casted to int because it returns a size_t type. You can use the right printf specifier instead of casting.

• The sdsempty() function creates an empty zero-length string:

  sds mystring = sdsempty();
  printf("%d\n", (int) sdslen(mystring));
  
  output> 0
  

• The sdsdup() function duplicates an already existing SDS string:

  sds s1, s2;
  
  s1 = sdsnew("Hello");
  s2 = sdsdup(s1);
  printf("%s %s\n", s1, s2);
  
  output> Hello Hello
  

Obtaining the string length

size_t sdslen(const sds s);

In the examples above we already used the sdslen function in order to get the length of the string. This function works like strlen of the libc except that:

• It runs in constant time since the length is stored in the prefix of SDS strings, so calling sdslen is not expensive even when called with very large strings.
• The function is binary safe like any other SDS string function, so the length is the true length of the string regardless of the content, there is no problem if the string includes null term characters in the middle.

As an example of the binary safeness of SDS strings, we can run the following code:

sds s = sdsnewlen("A\0\0B",4);
printf("%d\n", (int) sdslen(s));

output> 4

Note that SDS strings are always null terminated at the end, so even in that case s[4] will be a null term, however printing the string with printf would result in just "A" to be printed since libc will treat the SDS string like a normal C string.

Destroying strings

void sdsfree(sds s);

The destroy an SDS string there is just to call sdsfree with the string pointer. Note that even empty strings created with sdsempty need to be destroyed as well otherwise they'll result into a memory leak.

The function sdsfree does not perform any operation if instead of an SDS string pointer, NULL is passed, so you don't need to check for NULL explicitly before calling it:

if (string) sdsfree(string); /* Not needed. */
sdsfree(string); /* Same effect but simpler. */

Concatenating strings

Concatenating strings to other strings is likely the operation you will end using the most with a dynamic C string library. SDS provides different functions to concatenate strings to existing strings.

sds sdscatlen(sds s, const void *t, size_t len);
sds sdscat(sds s, const char *t);

The main string concatenation functions are sdscatlen and sdscat that are identical, the only difference being that sdscat does not have an explicit length argument since it expects a null terminated string.

sds s = sdsempty();
s = sdscat(s, "Hello ");
s = sdscat(s, "World!");
printf("%s\n", s);

output> Hello World!

Sometimes you want to cat an SDS string to another SDS string, so you don't need to specify the length, but at the same time the string does not need to be null term