FFV1 im AV Archivbereich
(nestor Webinar)

Peter Bubestinger-Steindl
(p.bubestinger@av-rd.com)

2021-05-20

FFV1: Was bringt’s? Was kann’s? Was kostet’s? Und wie geht’s weiter?

FFV1: Was bringt’s?

It saves disk space (*)

  • No generation loss, bit-proof encoding.
  • Master for generating different format (access) copies.
  • It “just works™”

    At highest quality.

    +Significant Properties = safe. long-term. open.

How it all began…?

Format consensus and interoperability:

Image source: https://vancouversun.com/news/metro/stanley-cup-riot-investigation-winds-down-as-last-two-suspects-charged
Image source: https://vancouversun.com/news/metro/stanley-cup-riot-investigation-winds-down-as-last-two-suspects-charged

2009, AT, Vienna

Hermann Lewetz at the Austrian Mediathek wanted to apply the same principles to video preservation as we all agree on the standards for audio:
Lossless or uncompressed, if possible. Lossy is to be avoided.

See: IASA-TC 03

Uncompressed, J2K, MXF: Okay. But tricky.

Image source: https://theawesomer.com/switches-the-secret-life-of-components/616632/
Image source: https://theawesomer.com/switches-the-secret-life-of-components/616632/

Options?

Lossless codec comparison graph (Source: MSU, 2007)
Lossless codec comparison graph (Source: MSU, 2007)

The format consensus was… in 2014!

Image source: http://www.preforma-project.eu/
Image source: http://www.preforma-project.eu/

FFmpeg & FFV1: Getting involved.

Image source: Peter B.
Image source: Peter B.

Plus: ©onflicts of Inte®est℠™

Image source: https://www.cumanagement.com/articles/2019/12/ransomware-threat
Image source: https://www.cumanagement.com/articles/2019/12/ransomware-threat

How dare you…

“…keeping us from wanting to do affordable, easy, open and interoperable lossless? For long-term reasons.”

Image source: https://www.debate.com.mx/mundo/Greta-Thunberg-dice-que-estan-subestimando-la-fuerza-de-jovenes-20191203-0092.html
Image source: https://www.debate.com.mx/mundo/Greta-Thunberg-dice-que-estan-subestimando-la-fuerza-de-jovenes-20191203-0092.html

FFV1: A short history about almost everything

  • 2003: FFV1 created in Open Source project “FFmpeg”
  • 2006: Bitstream frozen (version 1)
  • 2009: Picked up for preservation
  • 2010: Funding improvements (Mediathek)
  • 2012: Added 14bit RGB, Multithreading, SliceCRC
  • 2013: Official release of “FFV1.3”
  • 2014: PREFORMA Project
  • 2016: IETF Standardization in progress
  • 202+: Number of FFV1 users growing. Fast.

FFV1: 2021?

Practically the current defacto (and soon IETF) standard lossless preservation format.

And it’s Free and Open Source, so accessible and usable without artifical restrictions.

FFV1: Was kann’s?

Page 1

  • Absolutely completely reversible lossless compression.
  • Always lossless. Nothing to misconfigure.
  • Similar compression as J2K (but faster).
  • Not the fastest lossless codec, but best speed/size/support ratio.
  • …

Fast enough for SD editing in realtime

Smooth FFV1 playback in Virtualbox (Image source: Peter B.)
Smooth FFV1 playback in Virtualbox (Image source: Peter B.)

Quality (original = lossless)

Image source: Peter B.
Image source: Peter B.

Quality (x264 8Mbps)

Image source: Peter B.
Image source: Peter B.

Quality (x264 2Mbps)

Image source: Peter B.
Image source: Peter B.

Quality (x264 850k)

Image source: Peter B.
Image source: Peter B.

Quality (x264 200k)

Image source: Peter B.
Image source: Peter B.

FFV1: Was kann’s?

Page 2

  • Preserves almost all common “pix_fmts” as-is.
  • Open since day 1.
  • YUV + RGB up to 16bpc (*)
  • Widely supported by now (thx to FFmpeg libs).
  • Easy to migrate to other/future formats.
  • Error detection built-in (Slices)

Slices

Image source: Peter B. Example what 9 slices would look like.

Error detection/concealment

Image source: Peter B.
Image source: Peter B.

What is MKV?

“The Matroska Multimedia Container is a free, open-standard container format, a file format that can hold an unlimited number of video, audio, picture, or subtitle tracks in one file.”

Source: Wikipedia: Matroska

What is MKV?

  • It’s a container (like MOV, AVI, MXF)
  • Initial release: December 2002
  • IETF standardization in progress.
  • Also Open since day 1.
  • And btw: the base for Google’s “WebM” format

MKV is awesome.

  • Can contain/keep almost any source material (*)
  • Which is good for preservation
  • Like: any codecs, VBR, VFR, color infos, etc etc
  • Well documented and well supported (*)
  • It’s tech-design allows adaptability to future things.

FFV1 & MKV = ❤️

  • FFV1 goes well into: MKV, but also AVI/MOV.
  • All 3 combinations are preservation-approved.
  • With MKV being the most vendor-neutral and open/accessible option.
  • Not only stable, but future-proof.
  • Theoretically no format obsolescence issues.

Users: Moving to FFV1/MKV

Image source: British Film Institute (2019-07)   Image source: British Film Institute (2020-10)

British Film Institute: 2019 vs 2020

FFV1: Performance

Lossless codec comparison graph (Source: MSU, 2007)
Lossless codec comparison graph (Source: MSU, 2007)

FFV1: Performance

Source: Peter B.
Source: Peter B.

Dude, it’s huge!

PAL SD (720x576px)

Uncompressed: ca. 1.37 GB/m
(< 4 min. on a DVD)

  • FFV1 VHS: ca. 350 MB/min
  • FFV1 DigiBeta : ca. 700 MB/min

FFV1: Size estimations

Film

  • Varying between 1/3 and 2/3.
  • Average = 1/2 of uncompressed.
  • Better with less bits (=less noise).

Professional FFV1 Fun Facts!

  • Noise compresses bad (no pattern).
  • Synthetic (CGI) compresses best: ~1/9th (!)
  • Lossy is always smaller.

Thanks Alex for the Data!

Frisch aus dem Archiv der sozialen Demokratie Data: Archiv der sozialen Demokratie (2021-05), Graph: Peter B.

Hardware & Speed.

Data: Archiv der sozialen Demokratie (2021-05), Graph: Peter B.
Data: Archiv der sozialen Demokratie (2021-05), Graph: Peter B.

How to use it?

Virtualdub2 & FFV1 (Source: Peter B.)
Virtualdub2 & FFV1 (Source: Peter B.)

Most people however, prefer this:

$ ffmpeg -c:v ffv1 -c:a pcm_s16le -map 0 preserved_yo.mkv

(FFmpeg is king. But that’s another story.)

When to use it?

  • Analogue captures.
  • Normalizing (=converting) born-digital files with proprietary (non-whitelist) codecs.
  • Scientific video research data.
  • Personal originals.
  • More backups than uncompressed.
  • Network transfer.

When not to use it?

  • Born digital already (=lossy encoded) in “preservable” format.
    (Refer to the format whitelist approach)
  • If filesize simply is too much for your setup/budget to handle.
  • …or simply overkill for a given use case.
  • Or you prefer lossy artefacts.

The Whitelist approach

A format normalization decision for preservation:

  1. Does an open, FOSS licensed implementation for decoding (to uncompressed) exist?
  2. Is the format a documented and/or an open standard?
  3. Is the license/patent situation of the format (and codec implementations) resolved for unlimited access/usage?
  4. Can you put that in your AIP without future restrictions for access? (maybe has to run then on Unknown Future Alien Technology 3000™?)

Was kostet’s?

  • Computation time (CPU).
  • FOSS = Free to use, study, share & improve.
  • Support- and license costs: Optional.
    (Think of it like Roads and Bridges)

Rawcooked

“It’s like ZIP for film! Awesome.”®

  • 1-stop-shop for image sequences (🎞️) to FFV1/PCM in MKV. 📼 💾
  • And can completely bit-hashcode-proof restore the original file sequence and structure. 🥳
  • Has a Free + Paid license model. 🍒
  • Yes, it’s OpenSource. ✅

https://mediaarea.net/RAWcooked

Und wie geht’s weiter?

  • Finalize standardization.
  • Better support for easier color handling.
  • Continue improving tool/support landscape.

Und wie geht’s weiter?

Image source: https://en.wikipedia.org/wiki/Success_Kid
Image source: https://en.wikipedia.org/wiki/Success_Kid

Thanks to everyone who contributed!

An Original Drawing (Ireland, dated around 2014 AD)
An Original Drawing (Ireland, dated around 2014 AD)

Fersch.

Frasch’n?

(ffv1@av-rd.com)

Dave Rice and Peter B. (Image Source: Erwin Verbruggen (2016-07))
Dave Rice and Peter B. (Image Source: Erwin Verbruggen (2016-07))