Answers, Part 2
Now we are ready to perform some comparisons.
Comparing IP Addresses to Subnets
To recap, we can retrieve an IP address in bitset form via
Convert IP Address to Bitset
(
it
, (
left shift 24 of item 0 of it
+ left shift 16 of item 1 of it
+ left shift 8 of item 2 of it
+ item 3 of it
) of (
tuple string item 0 of it as integer as bit set
, tuple string item 1 of it as integer as bit set
, tuple string item 2 of it as integer as bit set
, tuple string item 3 of it as integer as bit set
) of concatenation ", " of substrings separated by "." of it
) of "192.168.10.33"
192.168.10.33, 11000000101010000000101000100001
Evaluation time: 0.140 ms
Evaluates to singular object of type ( string, bit set )
And we can retrieve a Subnet and Subnet Mask in bitset forms via
Convert Subnet CIDR to Bitsets
(
it
, (
(
item 0 of it
+ item 1 of it
+ item 2 of it
+ item 3 of it
) of (
left shift 24 of item 0 of it
, left shift 16 of item 1 of it
, left shift 8 of item 2 of it
, item 3 of it
) of (
tuple string item 0 of it as integer as bit set
, tuple string item 1 of it as integer as bit set
, tuple string item 2 of it as integer as bit set
, tuple string item 3 of it as integer as bit set
) of concatenation ", " of substrings separated by "." of preceding text of it
, (bit set (it)) of (
concatenation of "1" of integers in (1,(it))
& concatenation of "0" of integers in (31, it)
) of (it as integer) of following text of it
) of first "/" of it
) of "192.168.10.32/27"
192.168.10.32/27, ( 11000000101010000000101000100000, 11111111111111111111111111100000 )
Evaluation time: 0.230 ms
Evaluates to singular object of type ( string, ( bit set, bit set ) )
Now we wish to compare an IP address to a Subnet.
For this we will use the bitwise "AND" operator, which in Relevance is represented by the "*" symbol. In a bitwise AND, each bit position from the two values is compared. If there is a "1" in that position of both bitsets, the result for that bit is a "1". If either of the bitsets has a "0" in that position, the result is a "0". For example
010 (2)
AND 011 (3)
---
010 (2)
When comparing an IP address to a subnet, we AND every bit of the IP address against the Subnet Mask. If the result value is equal to the Subnet Address, then the IP is “in” that Subnet.
11000000 10101000 00001010 00100001 (IP address 192.168.10.33)
AND 11111111 11111111 11111111 11100000 (Subnet Mask 255.255.255.248)
-------- -------- -------- --------
11000000 10101000 00001010 00100000 (Subnet Address 192.168.10.32)
Because 192.168.10.33 AND 255.255.255.248 = the subnet address 192.168.10.32, this IP address is “in” the Subnet.
Combining all of that logic together, we can compare a list of subnet CIDR addresses to IP addresses via the following:
(
item 0 of it /* Subnet CIDR address */
, item 1 of it /* Readable IP address*/
, item 2 of it /* Subnet as bit set*/
, item 3 of it /* subnet mask as bit set */
, item 4 of it /* ip address as bit set */
, item 2 of it = item 3 of it * item 4 of it /* bool: subnet = IP address * subnet mask */
) of
(
item 0 of it
, item 1 of it
, (
( item 0 of it + item 1 of it + item 2 of it + item 3 of it) of
(
left shift 24 of item 0 of it
, left shift 16 of item 1 of it
, left shift 8 of item 2 of it
, item 3 of it
) of
(
tuple string item 0 of it as integer as bit set
, tuple string item 1 of it as integer as bit set
, tuple string item 2 of it as integer as bit set
, tuple string item 3 of it as integer as bit set
) of concatenation ", " of substrings separated by "." of preceding text of first "/" of item 0 of it
)
, (
(bit set (it)) of (concatenation of "1" of integers in (1,(it)) & concatenation of "0" of integers in (31, it)) of (it as integer) of following text of it
) of first "/" of item 0 of it /* Subnet CIDR list */
, (
(
item 0 of it
+ item 1 of it
+ item 2 of it
+ item 3 of it
) of
(
left shift 24 of item 0 of it
, left shift 16 of item 1 of it
, left shift 8 of item 2 of it
, item 3 of it
) of (
tuple string item 0 of it as integer as bit set
, tuple string item 1 of it as integer as bit set
, tuple string item 2 of it as integer as bit set
, tuple string item 3 of it as integer as bit set
) of concatenation ", " of substrings separated by "." of item 1 of it) /* Address list */
) of
( /* subnet list */
("192.168.10.32/27"; "192.168.128.0/27"; "192.168.128.64/27"; "192.168.128.128/27"; "192.168.128.192/26")
/* IP address list */
, ("192.168.10.33";"192.168.10.34";"192.168.10.35";"192.168.128.1")
)
192.168.10.32/27, 192.168.10.33, 11000000101010000000101000100000, 11111111111111111111111111100000, 11000000101010000000101000100001, True
192.168.10.32/27, 192.168.10.34, 11000000101010000000101000100000, 11111111111111111111111111100000, 11000000101010000000101000100010, True
192.168.10.32/27, 192.168.10.35, 11000000101010000000101000100000, 11111111111111111111111111100000, 11000000101010000000101000100011, True
192.168.10.32/27, 192.168.128.1, 11000000101010000000101000100000, 11111111111111111111111111100000, 11000000101010001000000000000001, False
192.168.128.0/27, 192.168.10.33, 11000000101010001000000000000000, 11111111111111111111111111100000, 11000000101010000000101000100001, False
192.168.128.0/27, 192.168.10.34, 11000000101010001000000000000000, 11111111111111111111111111100000, 11000000101010000000101000100010, False
192.168.128.0/27, 192.168.10.35, 11000000101010001000000000000000, 11111111111111111111111111100000, 11000000101010000000101000100011, False
192.168.128.0/27, 192.168.128.1, 11000000101010001000000000000000, 11111111111111111111111111100000, 11000000101010001000000000000001, True
192.168.128.64/27, 192.168.10.33, 11000000101010001000000001000000, 11111111111111111111111111100000, 11000000101010000000101000100001, False
192.168.128.64/27, 192.168.10.34, 11000000101010001000000001000000, 11111111111111111111111111100000, 11000000101010000000101000100010, False
192.168.128.64/27, 192.168.10.35, 11000000101010001000000001000000, 11111111111111111111111111100000, 11000000101010000000101000100011, False
192.168.128.64/27, 192.168.128.1, 11000000101010001000000001000000, 11111111111111111111111111100000, 11000000101010001000000000000001, False
192.168.128.128/27, 192.168.10.33, 11000000101010001000000010000000, 11111111111111111111111111100000, 11000000101010000000101000100001, False
192.168.128.128/27, 192.168.10.34, 11000000101010001000000010000000, 11111111111111111111111111100000, 11000000101010000000101000100010, False
192.168.128.128/27, 192.168.10.35, 11000000101010001000000010000000, 11111111111111111111111111100000, 11000000101010000000101000100011, False
192.168.128.128/27, 192.168.128.1, 11000000101010001000000010000000, 11111111111111111111111111100000, 11000000101010001000000000000001, False
192.168.128.192/26, 192.168.10.33, 11000000101010001000000011000000, 11111111111111111111111111000000, 11000000101010000000101000100001, False
192.168.128.192/26, 192.168.10.34, 11000000101010001000000011000000, 11111111111111111111111111000000, 11000000101010000000101000100010, False
192.168.128.192/26, 192.168.10.35, 11000000101010001000000011000000, 11111111111111111111111111000000, 11000000101010000000101000100011, False
192.168.128.192/26, 192.168.128.1, 11000000101010001000000011000000, 11111111111111111111111111000000, 11000000101010001000000000000001, False
Evaluation time: 3.328 ms
Evaluates to plural object of type ( string, string, bit set, bit set, bit set, boolean )
We can also add filtering, to retrieve the list of only the matching subnet & IP addresses via the following query.
For illustration purposes, I'm displaying the original subnet CIDR address, the original readable IP address, and the bit set representations for the Subnet, Subnet Mask, and IP address.
(
item 0 of it /* Subnet CIDR address */
, item 1 of it /* Readable IP address*/
, item 2 of it /* Subnet as bit set*/
, item 3 of it /* subnet mask as bit set */
, item 4 of it /* ip address as bit set */
) whose
(
item 2 of it = item 3 of it * item 4 of it /* bool: subnet = IP address * subnet mask */
) of
(
item 0 of it
, item 1 of it
, (
(
item 0 of it
+ item 1 of it
+ item 2 of it
+ item 3 of it
) of
(
left shift 24 of item 0 of it
, left shift 16 of item 1 of it
, left shift 8 of item 2 of it
, item 3 of it
) of
(
tuple string item 0 of it as integer as bit set
, tuple string item 1 of it as integer as bit set
, tuple string item 2 of it as integer as bit set
, tuple string item 3 of it as integer as bit set
) of concatenation ", " of substrings separated by "." of preceding text of first "/" of item 0 of it) /* Subnet CIDR List */
, (
(bit set (it)) of
(
concatenation of "1" of integers in (1,(it))
& concatenation of "0" of integers in (31, it)
) of (it as integer) of following text of it) of first "/" of item 0 of it
, (
(
item 0 of it
+ item 1 of it
+ item 2 of it
+ item 3 of it
) of (
left shift 24 of item 0 of it
, left shift 16 of item 1 of it
, left shift 8 of item 2 of it
, item 3 of it
) of (
tuple string item 0 of it as integer as bit set
, tuple string item 1 of it as integer as bit set
, tuple string item 2 of it as integer as bit set
, tuple string item 3 of it as integer as bit set
) of concatenation ", " of substrings separated by "." of item 1 of it /* IP Address list */
)
) of
( /* subnet list */
(
"192.168.10.32/27"
; "192.168.128.0/27"
; "192.168.128.64/27"
; "192.168.128.128/27"
; "192.168.128.192/26"
)
/* IP address list */
, (
"192.168.10.33"
; "192.168.10.34"
; "192.168.10.35"
; "192.168.128.1"
)
)
192.168.10.32/27, 192.168.10.33, 11000000101010000000101000100000, 11111111111111111111111111100000, 11000000101010000000101000100001
192.168.10.32/27, 192.168.10.34, 11000000101010000000101000100000, 11111111111111111111111111100000, 11000000101010000000101000100010
192.168.10.32/27, 192.168.10.35, 11000000101010000000101000100000, 11111111111111111111111111100000, 11000000101010000000101000100011
192.168.128.0/27, 192.168.128.1, 11000000101010001000000000000000, 11111111111111111111111111100000, 11000000101010001000000000000001
Evaluation time: 2.826 ms
Evaluates to plural object of type ( string, string, bit set, bit set, bit set )
Challenge Solution
Solving the original Relevance challenge is a matter of only returning the Subnet and IP address values where Address AND Subnet Mask = Subnet Address:
(
item 0 of it /* Subnet CIDR address in readable form */
, item 1 of it /* IP Address in readable form */
) of
(
item 0 of it /* Subnet CIDR address */
, item 1 of it /* Readable IP address*/
, item 2 of it /* Subnet as bit set*/
, item 3 of it /* subnet mask as bit set */
, item 4 of it /* ip address as bit set */
) whose
(
/* bool: subnet = IP address * subnet mask */
item 2 of it = item 3 of it * item 4 of it
) of
(
item 0 of it /* Subnet CIDR address in readable form */
, item 1 of it /* IP Address in readable form */
, (
(
item 0 of it /* Combine the octets of the Subnet Address bitsets into one bitset */
+ item 1 of it
+ item 2 of it
+ item 3 of it
) of
(
/* Shift the octet bitsets of the Subnet Address into their correct positions */
left shift 24 of item 0 of it
, left shift 16 of item 1 of it
, left shift 8 of item 2 of it
, item 3 of it
) of (
/* Split the octets of the Subnet Address into bitsets */
tuple string item 0 of it as integer as bit set
, tuple string item 1 of it as integer as bit set
, tuple string item 2 of it as integer as bit set
, tuple string item 3 of it as integer as bit set
) of concatenation ", " of substrings separated by "." of preceding text of first "/" of item 0 of it)
, (
/* Create a bitset of the Subnet Mask portion */
(bit set (it)) of (
concatenation of "1" of integers in (1,(it))
& concatenation of "0" of integers in (31, it)) of (it as integer) of following text of it
/* Split the Subnet CIDR Address on "/" - Address is before, Mask is after */
) of first "/" of item 0 of it
, (
(
/* Merge the octet bitsets for the IP Address via bitwise "OR" operator */
item 0 of it
+ item 1 of it
+ item 2 of it
+ item 3 of it
) of (
/* Shift the octet bitsets for IP Address into their correct positions */
left shift 24 of item 0 of it
, left shift 16 of item 1 of it
, left shift 8 of item 2 of it
, item 3 of it
) of (
/* Split the octets of the IP Address, and make a bitset from each octet */
tuple string item 0 of it as integer as bit set
, tuple string item 1 of it as integer as bit set
, tuple string item 2 of it as integer as bit set
, tuple string item 3 of it as integer as bit set
) of concatenation ", " of substrings separated by "." of item 1 of it)
) of
( /* subnet list */
(
"192.168.1.0/24"
; "192.168.2.0/25"
; "192.168.2.128/25"
; "192.168.3.0/25"
; "192.168.3.128/26"
; "192.168.3.192/26"
)
/* IP address list */
, (
"192.168.1.50"
; "192.168.2.16"
; "192.168.2.220"
; "192.168.3.10"
; "192.168.3.110"
; "192.168.3.224"
)
)
192.168.1.0/24, 192.168.1.50
192.168.2.0/25, 192.168.2.16
192.168.2.128/25, 192.168.2.220
192.168.3.0/25, 192.168.3.10
192.168.3.0/25, 192.168.3.110
192.168.3.192/26, 192.168.3.224
Evaluation time: 7.466 ms
Evaluates to plural object of type ( string, string )
Efficiency and Scaling
Those comparisons are nice, but do have a couple of limitations.
- Only the subnets that actually have a matching IP address are included
- The way we are creating cross-product of ( Every Subnet CIDR X Every IP Address) can be slow when run at scale
For every additional Subnet, we have to re-calculate the IP address. If there is one subnet and 10 IP addresses, I have to run the IP address calculation 10 times. If there are 2 subnets, double the IP address calculation to run 20 times. Add a third subnet and the IP address is calculated 30 times. In my use case, there is added overhead in that I’m retrieving the list of IP addresses via session relevance as well, so those lookups add overhead of their own.
For a primer on how this cross-product efficiency works, I’d recommend the excellent blog & post by @brolly33 at Efficient Session Relevance Query for Computer Properties
In the real data set, I'll be dealing with up to 14 thousand Subnets and up to a half million IP addresses. Each Subnet address is read from an Analysis in the "BES Asset Management" site, and each IP address is read from the properties of a BES Unmanaged Asset. Looking up these values repeatedly and re-calculating their bit set values can gets expensive quickly. Also in my use-case I need to display each Subnet Address, and the number of IP addresses that are matched on that subnet, so I need to preserve the Subnet Addresses that had 0 matches.
The way I try to make this more efficient is to lookup and calculate all of the subnet addresses, one time, and store the results in a string set. Then I lookup and calculate all of the IP addresses, one time, and store it in another string set. Then I loop through each Subnet in the set, compare each Subnet to each element of the IP Address set, and preserve the matching values.
To store these as Sets, we have to convert back-and-forth between Bit Sets and Strings, which is a bit more difficult to read. For the Subnets (which I’ll want to display back), I keep the original Subnet CIDR string, the subnet address in bit set format, and the original subnet mask in bit set format, all delimited by colons. For the IP addresses (which I don’t care to display back), I only keep the IP address in bit set format.
(
item 0 of it
, elements of item 1 of it
) of
(
elements of item 0 of it
, item 1 of it
) of
(
set of (
item 0 of it
& ":"
& /* Subnet as bit set*/ item 1 of it as string
& ":"
& /* subnet mask as bit set */ item 2 of it as string
) of
(
it
, (
(
item 0 of it
+ item 1 of it
+ item 2 of it
+ item 3 of it
) of
(
left shift 24 of item 0 of it
, left shift 16 of item 1 of it
, left shift 8 of item 2 of it
, item 3 of it
) of
(
tuple string item 0 of it as integer as bit set
, tuple string item 1 of it as integer as bit set
, tuple string item 2 of it as integer as bit set
, tuple string item 3 of it as integer as bit set
) of concatenation ", " of substrings separated by "." of preceding text of first "/" of it
)
, (
(bit set (it)) of
(
concatenation of "1" of integers in (1,(it))
& concatenation of "0" of integers in (31, it)
) of (it as integer) of following text of it) of first "/" of it
) of
(
( /* Subnet List */
"192.168.10.32/27"
; "192.168.128.0/27"
; "192.168.128.64/27"
; "192.168.128.128/27"
; "192.168.128.192/26"
)
)
, set of (it as string) of
(
(
item 0 of it
+ item 1 of it
+ item 2 of it
+ item 3 of it
) of
(
left shift 24 of item 0 of it
, left shift 16 of item 1 of it
, left shift 8 of item 2 of it
, item 3 of it
) of
(
tuple string item 0 of it as integer as bit set
, tuple string item 1 of it as integer as bit set
, tuple string item 2 of it as integer as bit set
, tuple string item 3 of it as integer as bit set
) of concatenation ", " of substrings separated by "." of it
) of
(
"192.168.10.33"
; "192.168.10.34"
; "192.168.10.35"
; "192.168.128.1"
)
)
192.168.10.32/27:11000000101010000000101000100000:11111111111111111111111111100000, 11000000101010000000101000100001
192.168.10.32/27:11000000101010000000101000100000:11111111111111111111111111100000, 11000000101010000000101000100010
192.168.10.32/27:11000000101010000000101000100000:11111111111111111111111111100000, 11000000101010000000101000100011
192.168.10.32/27:11000000101010000000101000100000:11111111111111111111111111100000, 11000000101010001000000000000001
192.168.128.0/27:11000000101010001000000000000000:11111111111111111111111111100000, 11000000101010000000101000100001
192.168.128.0/27:11000000101010001000000000000000:11111111111111111111111111100000, 11000000101010000000101000100010
192.168.128.0/27:11000000101010001000000000000000:11111111111111111111111111100000, 11000000101010000000101000100011
192.168.128.0/27:11000000101010001000000000000000:11111111111111111111111111100000, 11000000101010001000000000000001
192.168.128.128/27:11000000101010001000000010000000:11111111111111111111111111100000, 11000000101010000000101000100001
192.168.128.128/27:11000000101010001000000010000000:11111111111111111111111111100000, 11000000101010000000101000100010
192.168.128.128/27:11000000101010001000000010000000:11111111111111111111111111100000, 11000000101010000000101000100011
192.168.128.128/27:11000000101010001000000010000000:11111111111111111111111111100000, 11000000101010001000000000000001
192.168.128.192/26:11000000101010001000000011000000:11111111111111111111111111000000, 11000000101010000000101000100001
192.168.128.192/26:11000000101010001000000011000000:11111111111111111111111111000000, 11000000101010000000101000100010
192.168.128.192/26:11000000101010001000000011000000:11111111111111111111111111000000, 11000000101010000000101000100011
192.168.128.192/26:11000000101010001000000011000000:11111111111111111111111111000000, 11000000101010001000000000000001
192.168.128.64/27:11000000101010001000000001000000:11111111111111111111111111100000, 11000000101010000000101000100001
192.168.128.64/27:11000000101010001000000001000000:11111111111111111111111111100000, 11000000101010000000101000100010
192.168.128.64/27:11000000101010001000000001000000:11111111111111111111111111100000, 11000000101010000000101000100011
192.168.128.64/27:11000000101010001000000001000000:11111111111111111111111111100000, 11000000101010001000000000000001
Evaluation time: 3.528 ms
Evaluates to plural object of type ( string, string )
The comparison for my scalable query is as follows. For each Subnet element, I split the element on colons (":"), convert the subnet address and masks back from a string to a bit set, and compare it to each element of the IP address set (also converting that back from a string to a bit set). Then I keep a count of the number of IP addresses that matched the bit set.
(
preceding text of first ":" of item 0 of it /* Subnet Address in CIDR readable form */
, number of
(
/* For this Subnet, count the number of matching IP addresses in the IP Address Bitset string set */
item 1 of it /* Subnet Address Bitset */
, item 2 of it /* Subnet Mask Bitset */
, elements of item 3 of it /* Loop through each IP Address bitset string */
) whose (
/* Compare this IP Address to the Subnet */
/* IP Address Bitset AND Subnet Mask Bitset = Subnet Address Bitset */
bit set (item 2 of it) * item 1 of it = item 0 of it
)
) of
(
/* For each element of the Subnet Address Bitset, split out the Subnet CIDR Address, the Subnet Address Bitset, and the Subnet Mask Bitset */
item 0 of it
, bit set (preceding text of first ":" of following text of first ":" of item 0 of it)
, bit set (following text of first ":" of following text of first ":" of item 0 of it)
, item 1 of it /* Set of IP Address Bitset strings */
) of
(
/* Iterate through each element of the Subnet String Set */
elements of item 0 of it
, item 1 of it
) of
(
set of (
/* Create a delimited string set of Subnet CIDR:Subnet Address Bitset:Subnet Mask Bitset */
item 0 of it
& ":"
& /* Subnet as bit set*/ item 1 of it as string
& ":"
& /* subnet mask as bit set */ item 2 of it as string
) of
(
it /* Preserve the original subnet CIDR address in readable form */
, (
(
/* Combine the subnet octet bitsets into one subnet address bitset */
item 0 of it
+ item 1 of it
+ item 2 of it
+ item 3 of it
) of
(
/* Shift each subnet address octet bitset into their positions */
left shift 24 of item 0 of it
, left shift 16 of item 1 of it
, left shift 8 of item 2 of it
, item 3 of it
) of
(
/* Split the Subnet Address octets and convert each octet to a bitset */
tuple string item 0 of it as integer as bit set
, tuple string item 1 of it as integer as bit set
, tuple string item 2 of it as integer as bit set
, tuple string item 3 of it as integer as bit set
) of concatenation ", " of substrings separated by "." of preceding text of first "/" of it /* Subnet Addresses */
)
, (
/* Create a bitset of the subnet mask */
(bit set (it)) of
(
/* Create a string representing the subnet mask bits */
concatenation of "1" of integers in (1,(it))
& concatenation of "0" of integers in (31, it)
) of (it as integer) of following text of it /* Subnet CIDR mask */
) of first "/" of it
) of (
( /* Subnet CIDR Addresses */
"192.168.1.0/24"
; "192.168.2.0/25"
; "192.168.2.128/25"
; "192.168.3.0/25"
; "192.168.3.128/26"
; "192.168.3.192/26" )
)
/* Create a set of IP Address Bitset Strings - so we only retrieve/convert IP addresses one time */
, set of (it as string) of
(
(
(
/* Combine the IP Address octet bitsets via bitwise OR */
item 0 of it
+ item 1 of it
+ item 2 of it
+ item 3 of it
) of
(
/* Shift each IP address octet bitset into its correct position */
left shift 24 of item 0 of it
, left shift 16 of item 1 of it
, left shift 8 of item 2 of it
, item 3 of it
) of
(
/* Separate the octets of the IP address, and make a bitset of each octet */
tuple string item 0 of it as integer as bit set
, tuple string item 1 of it as integer as bit set
, tuple string item 2 of it as integer as bit set
, tuple string item 3 of it as integer as bit set
) of concatenation ", " of substrings separated by "." of it /* IP Addresses */
)
) of (
/* IP Address list */
"192.168.1.50"
; "192.168.2.16"
; "192.168.2.220"
; "192.168.3.10"
; "192.168.3.110"
; "192.168.3.224"
)
)
192.168.1.0/24, 1
192.168.2.0/25, 1
192.168.2.128/25, 1
192.168.3.0/25, 2
192.168.3.128/26, 0
192.168.3.192/26, 1
Evaluation time: 1.701 ms
Evaluates to plural object of type ( substring, bit set, bit set, integer )
Even at this very small scale, this query is much more efficient to execute.
I hope this challenge is interesting or useful to you at some point. Comparing IP addresses is itself a real-world use case, but other use cases come to mind; such as evaluating the UserAccountControl value of a Windows user account (where each bit of the value represents an attribute such as “Account Disabled”, “Password Change Required”, etc.) or in Registry values such as Meltdown / Spectre mitigations, where each Bit of the registry value represents a mitigation to enable or disable.