Monthly Archives: February 2019

Upload filters aka The Censorship machine

Since end of 2016 the European Parliament has filed a proposal for a directive in the area of digital markets and copyrights. As part of this proposal the Article 13 introduces a new concept :

Internet platforms hosting “large amounts” of user-uploaded content must monitor user behavior and filter their contributions to identify and prevent copyright infringement.

As you may imagine this changes the game pretty much.

Let’s make an example : a rightholder of music rights may ask platforms like www.soundcloud.com (Germany) to keep a look over a set of their works. Soundcloud will have to start monitoring all uploads to make sure that those materials are not uploaded by anyone on their platform.

Impact of this regulation, if it is going to pass, will be pretty strong on the EU contries economy. Let’s try to put down some points :

  1. Putting all the control burden on internet platforms hosting contents will probably result in :
    • being much more difficult for EU companies to compete with US/Asia content providers
    • get-away from EU countries for all new startups and existing companies in order to not have to comply with regulation

  2. Filter technology is too vast and complicated to be approached by each and single content provider : hundreds of rightholders requiring control over multiple sets of data ( text, images, audio, video, music score, software code ) will generate the need of content check providers that will de facto have censorship power .

  3. Guilty until proven innocent paradigma : if a filter erroneously blocks legal content it will be up to the content owner fight to make his content reinstated
  4. False positives : as in all automated checking procedures the number of false positives could be extremely high resulting in a limitation of freedom of expression

Many campaigns around this can be found :

  • Save Your Internet: “Stand up and ask Europe to protect Your Internet” (offers contact-your-MEP tool)
  • Say No to Online Censorship by the Civil Liberties Union for Europe: “Act now! It’s about our freedom to speak. It’s about censorship.” (offers email-your-MEP tool)
  • #SaveTheMeme,referring to parodies and other expressions of web culture that may be removed by such filtering technology
  • Create•Refresh: “These changes put the power of small, independent creators in jeopardy. Creative expression will effectively be censored, leaving only the bigger, more established players protected. Many of the sites that we use every day for information or entertainment may cease to exist.”
  • Save Codeshare

Thanks to Julia Reda (Pirate Party, EU Parliament member) for a lot of information on this topic.

Allocating memory inside a Varnish vmod


Writing varnish modules is pretty well documented by the standard varnish documentation, tutorials  and thanks to valuable work from other people here  . There are some areas I felt the need to be further clarified and this post tries to do that.

Allocating memory inside a vmod is tricky if you need to free it when the current Request is destroyed. Here are some ways :

  • per request memory allocation i.e. scope is the request lifetime so memory will be freed when the request is destroyed) :
void WS_Init(struct ws *ws, const char *id, void *space, unsigned len);
unsigned WS_Reserve(struct ws *ws, unsigned bytes);
void WS_MarkOverflow(struct ws *ws);
void WS_Release(struct ws *ws, unsigned bytes);
void WS_ReleaseP(struct ws *ws, char *ptr);
void WS_Assert(const struct ws *ws);
void WS_Reset(struct ws *ws, char *p);
char *WS_Alloc(struct ws *ws, unsigned bytes);
void *WS_Copy(struct ws *ws, const void *str, int len);
char *WS_Snapshot(struct ws *ws);
int WS_Overflowed(const struct ws *ws);
void *WS_Printf(struct ws *ws, const char *fmt, ...) __printflike(2, 3);

This is a per worker thread memory space allocation, no free necessary as data is removed when the request is detroyed. Ex :

VCL_STRING
vmod_hello(const struct vrt_ctx *ctx, VCL_STRING name)
{
   char *p;
   unsigned u, v;

   u = WS_Reserve(ctx->ws, 0); /* Reserve some work space */
   p = ctx->ws->f;         /* Front of workspace area */
   v = snprintf(p, u, "Hello, %s", name);
   v++;
   if (v > u) {
      /* No space, reset and leave */
      WS_Release(ctx->ws, 0);
      return (NULL);
   }
   /* Update work space with what we've used */
   WS_Release(ctx->ws, v);
   return (p);
}

Data is allocated starting with 64k and then when needed in 4k chunks in the cts->ws area. No varnish imposed limit.

  • (since varnish 4.0 up) Private Pointers : a way to have multi-scoped private data per each VCL, TASK. You may access private data either as passed on the VCL function signature or by calling directly VRT_priv_task(ctx, “name”) for example to obtain a per request place to hold :
    • free function
    • pointer to allocated data

This method is very interesting if you need a cleanup function to be called when the varnish request is destroyed.