Remove all the records from ets table having datestamp older than 10 seconds

Questions & Help Questions
,
1

I have an ets set table in an elixir app. I need to clean-up records which have their updated_at older than 10 seconds. is there a way I can set expirey or do manually without iterating over all the records(match records based on the timestamps greater than given time)

example record:

key: key_1
record: %{id: key_1, updated_at: ~N[2018-12-19 10:08:47.803075]}

so far I am able to have this code

def clean_stale(previous_key) do
  if previous_key == :"$end_of_table" do
    :ok
  else
    device = get(previous_key)
    next_key = :ets.next(__MODULE__, previous_key)
    if NaiveDateTime.diff(NaiveDateTime.utc_now, device.last_recorded_at) > 10 do
      remove(device.id)
    end
    clean_stale(next_key)
  end
end
2

You can try ets — OTP 29.0.2 (stdlib 8.0.1)

It would still iterate over the table, but unlike your current approach it wouldn’t copy any data from it. To make things easier, you might add an additional “column” to the table with updated_at as a timestamp for a simpler match statement.

3

How about if I make updated_at a unix timestamp. is there any method which will delete all records having updated_at greater than given timestamp? I mean this approach seems easy. I am not sure how pattern match would work though

4

How about if I make updated_at a unix timestamp. is there any method which will delete all records having updated_at greater than given timestamp?

Yes, that’s what I’m suggesting with :ets.match_delete/2. Let me try it locally with some fake data.

5

You are absolutely right. But I am newbie to ets concept. I googled and see syntax like this in examples

ets:match_delete(__MODULE__, [{['_', '$1', '_', '_'], [{'>', '$1', 2}], []}])

How exactly match would like if given timestamp is 1545215591

and thanks for your really quick answer

6

Probably

:ets.match_delete(__MODULE__, [{[:_, :"$1", :_, :_], [{:>, :"$1", timestamp}], []}])

if the timestamp is in the second column.

7 
    iex(13)> :ets.tab2list(table)
[
  {"test_dev!ce2",
  %DealorApiGateway.DeviceState.Device{
    id: "test_dev!ce2",
    last_recorded_at: 1545216134,
    switch_1: false,
    switch_2: false,
    switch_3: true
  }},
  {"test_dev!ce",
  %DealorApiGateway.DeviceState.Device{
    id: "test_dev!ce",
    last_recorded_at: 1545216126,
    switch_1: false,
    switch_2: false,
    switch_3: true
  }}
]

and I get this error

            iex(20)> :ets.match_delete(table, [{['_', '_', '$1', '_', '_', '_'], [{'>', '$1', timestamp}], [true]}])
** (ArgumentError) argument error
    (stdlib) :ets.internal_select_delete(DealorApiGateway.DeviceState, [{[{['_', '_', '$1', '_', '_', '_'], [{'>', '$1', 1545215591}], [true]}], [], [true]}])
    (stdlib) ets.erl:394: :ets.select_delete/2
    (stdlib) ets.erl:753: :ets.match_delete/2
8

Seems like you only have two columns in your ets table, a key and the corresponding struct. And the pattern [:_, :_, :"$1", :_, :_, :_] expects 6. I’d add a third column with the timestamp, and use [:_, :_, :"$1"]. Also note that '_' in erlang is an atom :_ in elixir (I’ve updated my response above).

9

Sorry. I was thinking it will go over all the values of a map.
Here is a complete set of data which I am trying

timestamp = 1545215591
device = %{
  switch_1: false,
  switch_2: false,
  switch_3: true,
  last_recorded_at: DateTime.to_unix(DateTime.utc_now),
  id: "test_dev!ce"
}

device_2 = %{
  switch_1: false,
  switch_2: false,
  switch_3: true,
  last_recorded_at: DateTime.to_unix(DateTime.utc_now),
  id: "test_dev!ce2"
}

:ets.new(:table, [:named_table])
:ets.insert(:table, {device.id, device, device.last_recorded_at})
:ets.insert(:table, {device_2.id, device_2, device_2.last_recorded_at})
:ets.tab2list(:table)
:ets.match_delete(table, [{['_', '_', '$1', '_', '_', '_'], [{'>', '$1', timestamp}], [true]}])
10

@idi527 I have updated how I insert data. and from your above answer, I have tried this

:ets.match_delete(__MODULE__, [{[:_, :_, :"$1", :_], [{:>, :"$1", timestamp}], []}])

and

:ets.match_delete(__MODULE__, [{[:_, :"$1", :_, :_], [{:>, :"$1", timestamp}], []}])

both seems not to be working

I have also tried

:ets.match_delete(__MODULE__, [{[:_, :_, :"$1"], [{:>, :"$1", timestamp}], []}]) 
:ets.match_delete(__MODULE__, [{[:_, :"$1", :_], [{:>, :"$1", timestamp}], []}])

Assuming first param is key, second is data and last is timestamp

11

Ah, it seems like you need to use select_delete to be able to use guards in the match spec.

:ets.insert(:table, {device.id, device.last_recorded_at, device})
:ets.insert(:table, {device_2.id, device_2.last_recorded_at, device_2})
:ets.select_delete(:table, [{{:_, :"$1", :_}, [{:>, :"$1", timestamp}], [true]}])

works for me.

12

Thanks. it worked

13

You might want to take a look at GitHub - ericmj/ex2ms: :ets.fun2ms for Elixir, translate functions to match specifications · GitHub to make those match specs more readable.

14

Also don’t forget :ets.fun2ms/1 to create matchspecs. Your example could be written as:

EDIT As @sorentwo correctly points out fun2ms relies on erlang parse transforms and will not work outside the shell.

15

I thought fun2ms relied on an Erlang macro? It works in iex but requires modified compilation to work outside the console. That’s why the project Nicd linked above exists (GitHub - ericmj/ex2ms: :ets.fun2ms for Elixir, translate functions to match specifications · GitHub), right?

16

Correct! My bad. I sloppily tested in iex. I’ve updated my post

17

Depending on what you’re using ets for you might want to try out cachex. It will let you expire keys based on when it was cached easily