Decoding JSON_ENCODED_TYPE_Data in Agenda's file format

Shortcuts
1

Hi, I have a few hundred notes in Agenda and want to export them into a custom data format. The actual target data format is not relevant here, but I’d like to include as much data and metadata as possible.

One of the important pieces of metadata to me is the various dates stored with a note.

Agenda’s export file format is quite easy to understand (thanks for this, much appreciated!). If you export your notes as an Agenda File, you get an .agenda file that is really a ZIP archive. After unzipping, there’s a Data.json inside that contains all the note data, plus a directory for attached files.

I started writing Swift code that can read the Data.json file into my custom data structures (using Decodable). So far, so good, it’s working pretty nicely already, but I have trouble understanding some of the data in the JSON file:

timeZoneIdentifier

timeZoneIdentifier is sometimes encoded as a plain string (e.g. “Europe/Amsterdam”), but sometimes it looks like this in the JSON:

"timeZoneIdentifier" : [
  "JSON_ENCODED_TYPE_Data",
  "8\/P0UQjwcXR29fD3cXENUtBVMDc0NHQyNTXQNXE1cNE1cXG21LU0NjPRtbB0dTQ0dXJyMXWxAAA="
]

I believe the string is base-64-encoded binary data, but I don’t understand the binary data format that’s encoded here. I tried decoding it as a (binary) plist or an NSTimeZone value using NSKeyedUnarchiver, but no success.

Date values

Similarly, date values are sometimes encoded like this (which I can decode easily):

"endDate" : [
  "JSON_ENCODED_TYPE_Date",
  "1589148000.0"
]

But sometimes date values are also encoded as opaque binary data that I don’t know the format of:

"startDate" : [
  "JSON_ENCODED_TYPE_Data",
  "8\/P0UQjwcXR29fD3cXENUtBVMDc0NHQyNTXQNXE1cNE1cXG21LU0NjPRtbB0dTQ0dXJyMXWxAAA="
]

Others

There are a few other keys that use similar JSON_ENCODED_TYPE_Data encodings, but I haven’t looked at those yet.

Any pointers on what these binary blobs contain and how they can be decoded would be much appreciated. I understand if you don’t want to divulge any secrets of your implementation, but I’d love it if the export data format was truly open to allow us to process our data in whatever form we want. Thank you!

PS: If there’s interest I’m open to sharing my code once I got it working. I can’t promise it supports every Agenda feature (only the ones I use in my notes) and I’m not planning on maintaining it, but it can be a starting point for your own experiments.

2

I think I got it (sleeping on a problem really helps!). What I didn’t notice yesterday (:man_facepalming:) is that all 2,000+ occurrences of this "JSON_ENCODED_TYPE_Data" in my sample data have the identical value of "8/P0UQjwcXR29fD3cXENUtBVMDc0NHQyNTXQNXE1cNE1cXG21LU0NjPRtbB0dTQ0dXJyMXWxAAA=".

Since this value is used for different data types (timeZoneIdentifier, startDate, endDate, eventProperties, eventIdentifier, templateProperties), I’m guessing this value is the encoding for nil/null/missing value.

I was able to verify in my sample data that:

  1. Notes either have a valid startDate and endDate, or both of these fields are “null”.
  2. The notes where startDate == endDate == null are indeed those that have no assigned date in Agenda (as far as I can tell from the subsample I checked).

I’m still not sure why a significant number of notes (~30% in my sample data) have a null timeZoneIdentifier.

Edit: I removed the backslash that escaped the forward slash in the JSON string. Some JSON libraries add this, but the backslash is not part of the string.

3

Bingo, you got it correctly. Indeed it’s a nil-placeholder. The reason is that our datamodel needs these to discriminate them between a “never been set” and a “was one time set but now deleted” scenario. The value of the nil placeholder will always be the same.

Any pointers on what these binary blobs contain and how they can be decoded would be much appreciated. I understand if you don’t want to divulge any secrets of your implementation, but I’d love it if the export data format was truly open to allow us to process our data in whatever form we want.

Just for the avoidance of any doubts, we don’t have any secrets regarding the data format, we on purposely picked a json like format so that others could parse it. We just didn’t have the time or needs so far to do extensive documentation, but happy to answer any questions you have.

PS: If there’s interest I’m open to sharing my code once I got it working. I can’t promise it supports every Agenda feature (only the ones I use in my notes) and I’m not planning on maintaining it, but it can be a starting point for your own experiments.

Would be great if you could indeed share for other users to continue and/or benefit from!

1 Like
4

Thanks a lot for confirming, @mekentosj.

I’m glad to hear it. I’m sure I’ll have more questions as I progress.

1 Like