MOON
Server: Apache
System: Linux server30c.hostingraja.org 3.10.0-962.3.2.lve1.5.63.el7.x86_64 #1 SMP Fri Oct 8 12:03:35 UTC 2021 x86_64
User: jibhires (1887)
PHP: 8.1.30
Disabled: show_source, system, shell_exec, passthru, exec, popen, proc_open, allow_url_fopen, symlink, escapeshellcmd, pcntl_exec
Upload Files
File: //opt/imunify360/venv/lib/python3.11/site-packages/im360/model/__pycache__/firewall.cpython-311.pyc
�

fn#g����dZddlZddlZddlZddlZddlmZddlmZddl	m
Z
ddlmZmZddlm
Z
ddlmZmZdd	lmZmZmZmZmZmZmZmZmZmZdd
lmZddlmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-ddl.m/Z/dd
l0m1Z1ddl2m3Z3m4Z4ddl5m6Z6ddl7m8Z8m9Z9m:Z:ddl;m<Z<ddl=m>Z>m?Z?m@Z@mAZAmBZBmCZCmDZDddlEmFZFmGZGmHZHeIZJejKeL��ZMeBed����dZNeBed����dZOeHeFjPjQZReHeFjSjQZTdee3deeIeIeIffd�ZUdee3deeIeIeIffd�ZVdee3deeIeIeIffd�ZWd�ZXd�ZYd�ZZd�Z[d e\fd!�Z]Gd"�d#e��Z^Gd$�d%e_e��Z`Gd&�d'e3��ZaGd(�d)e3��ZbGd*�d+e3��ZcGd,�d-e3��ZdGd.�d/e3��ZeGd0�d1e3��ZfGd2�d3e3��ZgGd4�d5e3��ZhGd6�d7e3��ZiGd8�d9e3��ZjGd:�d;e3��ZkGd<�d=e3��ZldS)>z,DB tables related to firewall functionality.�N)�	timedelta)�Enum)�reduce)�IPv4Network�IPv6Network)�starmap)�ior�
itemgetter)
�Any�Dict�Iterable�Iterator�List�Optional�Sequence�Tuple�Type�Union)�Signal)�JOIN�SQL�BooleanField�Case�	CharField�Check�CompositeKey�DoesNotExist�
FloatField�ForeignKeyField�IntegerField�PrimaryKeyField�	TextField�fn�prefetch)�
model_to_dict)�Reject)�Model�instance)�ApplyOrderBy)�CHUNK_SIZE_SQL_QUERY�split_for_chunk�timeit)�Country)�ALL�TCP�UDP�	IPNetwork�pack_ip_network�unpack_ip_network�is_net)�IP�	IPVersion�NumericIPVersionz
0.0.0.0/32�z::/64�model�
packed_ip_netc�b�|\}}}td||f��|j|kz|j|kzS)z�
    Filters ip addresses/networks contained in ip network net.

    :param model: model to apply query
    :param packed_ip_net: tuple of integers
    :return: peewee expression
    �(network_address & ?) == ?�r�netmask�version�r9r:�net�maskr?s     �I/opt/imunify360/venv/lib/python3.11/site-packages/im360/model/firewall.py�_filter_ip_net_subnetsrDHsE��'��C��w��(�4��+�6�6��=�D� �	"��=�G�#�	%��c�b�|\}}}td||f��|j|kz|j|kzS)z�
    Filters ip addresses/networks contained in ip network net.
    Does not includes network itself
    :param model: model to apply query
    :param packed_ip_net: tuple of integers
    :return: peewee expression
    r<r=r@s     rC� _filter_ip_net_subnets_exclusiverGZsE��'��C��w��(�4��+�6�6��=�4��	!��=�G�#�	%�rEc�`�|\}}}td|f��|j|kz|j|kzS)z�
    Filters ip addresses/networks that includes provided
    ip address/network, including network itself
    :param model: model to apply query
    :param packed_ip_net: tuple of integers
    :return: peewee expression
    z (? & netmask) == network_addressr=r@s     rC�_filter_ip_net_supernetsrIlsC��'��C��w��.���7�7��=�D� �	"��=�G�#�	%�rEc��	tj|��}t|��\}}}|js.|j�|��t
||||f��zS|j�|��t
||||f��zt||||f��zS#t$r|j�|��cYSwxYw�N)	�	ipaddress�
ip_networkr2�hostmask�ip�containsrIrG�
ValueError)r9�ip_str�ip_netrArBr?s      rC�_ip_search_conditionrT~s�����%�f�-�-��-�V�4�4���T�7���		��8�$�$�V�,�,�/G���T�7�+�0�0��
�
��!�!�&�)�)�*�5�3��g�2F�G�G�H�2�5�3��g�:N�O�O�P�
���)�)�)��x� � ��(�(�(�(�(�)���s�B � $C�Cc�v�t|��\}}}t||||f��t||||f��zSrK)r2rIrG)r9rSrArBr?s     rC�_net_search_conditionrV�sO��(��0�0��C��w�#�
��T�7�#���(���d�G�0D�E�E�F�FrEc�v��t�fd�dD����r;t�d�d�d��}tj|���d<nad�vr]tj�d��}t|��\}}}��|||d���tj|���d<�S)Nc3� �K�|]}|�vV��	dSrK�)�.0�k�kwargss  �rC�	<genexpr>z#_add_ip_net_args.<locals>.<genexpr>�s'�����
J�
J�1�1��;�
J�
J�
J�
J�
J�
JrE)�network_addressr>r?r^r>r?rO)�allr3r5�ip_net_to_string�adopt_to_ipvX_networkr2�update)r\rOrArBr?s`    rC�_add_ip_net_argsrc�s����
�
J�
J�
J�
J� I�
J�
J�
J�J�J�/�
��$�%�v�i�'8�&��:K�
�
���*�2�.�.��t���	
����
�
%�f�T�l�
3�
3��,�R�0�0���T�7��
�
� #���I�I�	
�	
�	
��*�2�.�.��t���MrEc��t|�d��ttf��r%t	|d��\|d<|d<|d<|d=|S)NrOr^r>r?)�
isinstance�getrrr2)�argss rC�_replace_ip_with_packed_reprrh�s\���$�(�(�4�.�.�;��"<�=�=��

�D��J�'�'�		
��"�#���O���O���J��KrE�returnc�P�|tjko|tj��kS)zWhether expiration time passed.)�IPList�NEVER�time��
expirations rC�
is_expiredrp�s�����%�C�*��	���*C�CrEc�"�eZdZdZdZdZdZdZdS)�
ActionTypezWhat to do with matching IPs.�drop�captcha�splashscreen�ignoreN)�__name__�
__module__�__qualname__�__doc__�DROP�CAPTCHA�SPLASHSCREEN�IGNORErYrErCrrrr�s*������'�'��D��G�!�L�
�F�F�FrErrc�T�eZdZdZdZeZdZeZdZeZ	dZ
e
Zd�Ze
defd���Zd	S)
�Purposez�IPList's purposes understood by the agent.

    An analog of i360.model.firewall.ActionType but for the new
    (DEF-17989) server sync case.

    �whitersrurtc��|jSrK)�value��selfs rC�__str__zPurpose.__str__�s
���z�rE�purposec���tjtjtjtjtjtjtjtjit|SrK)	r��WHITErkr{�BLACKr|�GRAYr}�GRAY_SPLASHSCREEN)�clsr�s  rC�listnamezPurpose.listname�sC��
�M�6�<��L�&�,��O�V�[�� �&�":�	
�
�'�
��	rEN)rwrxryrzr�r�r{rsr}rur|rtr��classmethod�strr�rYrErCr�r��s~��������
�E��E��D��D�!�L��L��G��G������s�����[���rEr�c��	��eZdZdZdZdZdZdZdZeeeefZ	e
eee	������Z
dZdZd	ZeZeZd
ZGd�d��Zed
���Zed
ed�d�e	������g���ZdZedd���Zed���Z edd����Z!ed���Z"ed���Z#edd���Z$e%d
d
���Z&e%d
d���Z'e%d���Z(e%dd
���Z)ed
���Z*ed
���Z+ed
���Z,edede�de�d���g���Z-Gd�d��Z.Gd�d��Z/e0de1e2de2fd ���Z3e0d!e2de2fd"���Z4e0d#e1e5fd$���Z6e0d#e1e5de7fd%���Z8e9de:e2e7ffd&���Z;e9		d_d'e<d(e=e2d)e2d*e7d+e>f
d,���Z?e9�fd-���Z@e9	d`d'eAe2eBeCfd!e=e2d/e>fd0���ZDe9d1���ZEe9d2���ZFe9		dad!e2d'e<fd3���ZGe9						dbd4���ZHe9d5e=e2fd6���ZIe9d7e=e<fd8���ZJe9d.d.d.eK��fd5e=e2fd9���ZLe9d`d:���ZMe9dad;���ZNe9dad<���ZOe9ed.fdePe<fd=���ZQe9d>���ZRe9d?���ZSe9d`d@���ZTdA�ZUe9dB���ZVdC�ZWdD�ZXd`dE�ZYe9d'e<de1e2fdF���ZZe9dG���Z[e9dH���Z\e]dI���Z^e9		dad.d.dJ�d'eAe_jBe_jCfd!e=e2dKe7fdL���Z`e9dM���Zae9		dadNebe<ebe2ecffd5e1e=fdO���Zde9d7ebe<ebe2ecfffdP���Zee9d7ebe<ebe2ecfffdQ���Zfe9d.d.dR�d'e<d!e=e2fdS���Zge9�fdT���Zhe9�fdU���Zie9�fdV���Zje9	d`d
d.d.dW�d'e<d!e=e2d/e>de=e:e<e2e7ffdX���Zke9d!e2dele2fdY���Zme9d!e2dele2fdZ���Zne9d`d'e<d!e2fd[���Zoe9d\e=ebfd]���Zpe9d7ePe:e<ebfde=e:e<e2ffd^���Zq�xZrS)crkz1The main persistent storage for various IP lists.�action_typer�r�r�r��local�groupr~�2c�b�eZdZdZe��Ze��Ze��Ze��ZdS)�IPList.Signalsz[Signals to inform subscribers about changes.

        Sender of event is listname.
        N)	rwrxryrzr�added�deleted�cleared�updatedrYrErC�Signalsr�
sL������	�	�������&�(�(���&�(�(���&�(�(���rEr�F��nullzlistname in ('{}')z','�r��constraintsrT��defaultr�c�B�ttj����SrK)�intrmrYrErC�<lambda>zIPList.<lambda>0s��3�t�y�{�{�#3�#3�rE�r�r��
country_id�r��column_namezscope in ('�')c�@�eZdZejZdZedddd��ZdZ	dS)�IPList.Meta�iplistr^r>r?r��residentN�
rwrxryr(�db�database�db_tabler�primary_key�schemarYrErC�Metar�Ys;�������;����"�l��y�)�Z�
�
�����rEr�c�$�eZdZed���ZdS)�IPList.OrderByc�P�ttjdd��tjfS)N))rr8r)rrkrorYrErCrozIPList.OrderBy.expirationbs����)�9�a�8�8�&�:K�K�KrEN)rwrxry�staticmethodrorYrErC�OrderByr�as2������	�	L�	L�
��	L�	L�	LrEr�ric
�2�tjjtjtjjtjtjjtjtj	jtj	dtji�
|tj��S)z�Given `action_type` string return corresponding list name.

        Return :attr:`GRAY` for an unknown/missing `action_type`.
        N)rrr{r�rkr�r|r�r}r�r~rf)r�s rC�action_type2listnamezIPList.action_type2listnamefs\��
�O�!�6�<���$�f�k��#�)�6�+C���#�V�]��&�+�
��#�k�6�;�
'�
'�
	(rEr�c��tjtjjtjtjjtjtjji|S)z0Return action_type corresponding to iplist name.)	rkr�rrr{r�r�r|r�r}�r�s rC�listname2action_typezIPList.listname2action_typets?��
�L�*�/�/��K��+�1��$�j�&=�&C�
��	�	rE�
propertiesc�x�t�|�|�tj��nd��S)zzGet iplist name corresponding to properties' action_type.

        Return GRAY for an unknown/missing action_type
        N)rkr�rf�ACTION_TYPE�r�s rC�get_listname_fromzIPList.get_listname_from}s<���*�*��%�
�N�N�6�-�.�.�.��
�
�	
rEc�^�|r |�dtj��ntjS)zSGet expiration from properties

        Return IPList.NEVER if property not definedro)rfrkrlr�s rC�get_expiration_fromzIPList.get_expiration_from�s+���
�J�N�N�<���6�6�6���	
rEc�r�|�|��}|jD]\}}||kr||fcS�Jd���)z#Return tuple listname and priority.rzcan't happen)r��IP_LIST_PRIORITIES)r�r��	_listname�priorityr�s     rC�get_listname_with_priority_fromz&IPList.get_listname_with_priority_from�s\���)�)�*�5�5�	�"%�"8�	*�	*��H�h��9�$�$���)�)�)�)�%� �.� � �qrErO�src�destro�full_accessc���||j|jfvs
Jd���t|��\}}}d�t�tjtj�����tj	�
|��tj|ktj|ktj
|k��D��\}	t����tj	�
|��tj|ktj|ktj
|ktj|	k�����t�|||dd����tj	�
|����}
|
�tj|ktj|ktj
|k��}
|
���}|D]#}|jj�||����$|jj�||�|������|S)aMove ip from src lists to dest list,
        as `move` used only in UI and CLI we add manual=True

        :param ip: ip address
        :param src: src lists (WHITE/BLACK/GRAY/GRAY_SPLASHSCREEN)
        :param dest: dst list (WHITE/BLACK)
        :param expiration: IPs TTL. 0 means permanent
        :param full_access: access to all ports
        :return int: items moved
        z"Move to GRAY list is not supportedc��g|]	}|j��
SrYrn)rZ�recs  rC�
<listcomp>zIPList.move.<locals>.<listcomp>�s*��
�
�
��
�N�
�
�
rETF)r�ror��manual�captcha_passed�rO)r�r�r2rk�selectr#�MAXro�wherer��in_r^r>r?�delete�executerbr�r��sendr�rf)
r�rOr�r�ror�rArBr?�max_expiration�q�rvr�s
             rC�movezIPList.move�s��(��H��!�
�
�
�
�0�
�
�
�
-�R�0�0���T�7�
�
��}�}�R�V�F�,=�%>�%>�?�?�E�E���#�#�C�(�(��&�#�-���$�&���'�)�	��
�
�
���	�
�
������O����$�$��"�c�)��N�d�"��N�g�%����/�	
�	
��'�)�)�)��M�M��!�#�� �
�
�
��%���#�#�C�(�(�
)�
)�
	
�
�G�G��"�c�)��N�d�"��N�g�%�
�
��
�Y�Y�[�[���	6�	6�H��K��$�$�X�"�$�5�5�5�5������t����2�����7�7�7��	rEc���tt|��jdit|����}|jj�|j|���|S)�`
        :param kwargs:
        :raises: IntegrityError
        :return: model instance
        r�rY)�superrk�creatercr�r�r�r��r�r\�inst�	__class__s   �rCr�z
IPList.create�sU���)�u�V�S�!�!�(�D�D�+;�F�+C�+C�D�D�������t�}���6�6�6��rENr�c��|�td���t����tj�|����}|�|�|j|k��}t|t��r$|�tj	|k��}nTt|��\}}}|�tj|ktj|ktj
|k��}|���}|r&|D]#}	|jj�|	|����$|S)zQDelete ip from lists if exists
        Return number of deleted records.
        Nzlistname should not be Noner�)rQrkr�r�r�r�r�rer�rOr2r^r>r?r�r�r�r�)
r�rOr�r�r�rArB�ver�rows_deleted�lsts
          rC�delete_from_listzIPList.delete_from_list�s�����:�;�;�;��M�M�O�O�!�!�&�/�"5�"5�h�"?�"?�@�@��������
�f�,�-�-�A��b�#���	�����	�R��(�(�A�A�,�R�0�0�N�C��s�����&�#�-���$�&���#�%���A�
�y�y�{�{���	5��
5�
5����#�(�(���(�4�4�4�4��rEc��t����tj�|�������}|r$|D]!}|jj�|���"|SrK)	rkr�r�r�r�r�r�r�r�)r��	listnames�num_deletedr�s    rC�clean_listszIPList.clean_listssy��
�M�M�O�O�!�!�&�/�"5�"5�i�"@�"@�A�A�I�I�K�K�	��	3�%�
3�
3����#�(�(��2�2�2�2��rEc���tj��t|������z
}|j|jk|�|��z}|j|jk|�|��z}|j|jk|jdkz|�|��z}|�	���
||z|z�����}|S)z�
        Removes obsoleted graylist/splashscreen+blacklist[manual=False] IPs.
        :param num_days: expired more than num_days ago
        :return: int rows deleted
        )�daysF)rmr�
total_secondsr�r�rpr�r�r�r�r�r�)r��num_days�
expiration_ts�graylist_ip_is_expired�graysplash_ip_is_expired�blacklist_ip_is_expiredr�s       rC�cleanup_expired_from_bglistz"IPList.cleanup_expired_from_bglists����	���i�X�&>�&>�&>�&L�&L�&N�&N�N�
�"%�,�#�(�":�c�n�n��?
�?
�"
��

�L�C�1�1��N�N�=�)�)�$*� �
�\�S�Y�
&��z�U�"�
$��n�n�]�+�+�
,�	 �
�J�J�L�L�
�U�&�*�+�)�*���
�W�Y�Y�	��rEc�T�|���}|r||j|kz}|rHt|��\}}}|tj|ktj|kztj|kzz}|����|���	��}|SrK)
rpr�r2rkr^r>r?r�r�r�)r�r�rO�clausesrArBr?r�s        rC�delete_expiredzIPList.delete_expired1s����.�.�"�"���	0��s�|�x�/�/�G�
�	�!0��!4�!4��C��w���'�3�.��>�T�)�+��>�W�,�.�
�G��*�*�,�,�$�$�W�-�-�5�5�7�7���rEc�n��t|t��sJ�|��fd�|D��}t�tjtjtjtjtjtj	tj
tjtjtj
tjtjtjtjt%jtj�j���d�����t0t2jtjt0jk����tj�|�����t������tj��}|�|� |��}|�|�!|��}|�|��j|k��}|rEtE|��}	|	r	tF|	fntH|f\}
}	|�|
�|	����}|r#|�t0j%|k��}|�2|�tj
�&|����}|S)Nc�4��g|]}|�j�jfv�|��SrY)r�r�)rZ�lnr�s  �rCr�z'IPList._fetch_query.<locals>.<listcomp>Ts8��������c�h��(=�>�>�>��>�>�>rE�scope��on)'re�listrkr�rOr�ro�
imported_from�ctime�deep�comment�countryr�r��auto_whitelistedr^r>r?r#�ifnullr�SCOPE_LOCAL�alias�joinr-r�
LEFT_OUTER�idr�r�rp�order_by�group_by�havingr4rVrT�coderP)r�r�rr�by_ip�by_country_code�
by_commentr�r�rA�search_conditions`          rC�_fetch_queryzIPList._fetch_queryEs,����)�T�*�*�*�*�*��!�����#����I�
�M�M��	����!��$����������
��"��'��&������	�&�,���8�8�>�>�w�G�G�
�
�"�T�'�4�?���'�*�0L�T�
N�
N�
�U�6�?�&�&�y�1�1�
2�
2�
�U�F�%�%�'�'�'�
(�
(�
�X�f�i�
 �
 �+	
�2���
�
�8�$�$�A������� � �A�������
�f�,�-�-�A��	4���-�-�C��3�&��,�,�*�E�2�
"��c�
���(�(��c�2�2�3�3�A��	9�������7�8�8�A��!������/�/�
�;�;�<�<�A��rEr�c�B�|j|fi|�����SrK�r�count)r�r��filter_argss   rC�fetch_countzIPList.fetch_count�s)���s��	�9�9�[�9�9�?�?�A�A�ArE�ipsc��t����tjtjk��}g}t|t
tdz����D]�}g}|D]Z}t|��\}}}	|�	tj
|ktj|kztj|	kz���[tt|��}
|t|�|
����z
}��|S)z�
        cannot use 'in' operator, since peewee's @hybrid_property can not
        compute lazy expression for 'IPList.ip_network' property
        �)rkr�r�r�SCOPE_GROUPr+r�r*r2�appendr^r>r?rr	r)r�rr��result�chunk�expressionsrOrArBr?�clauses           rC�fetch_for_group_synczIPList.fetch_for_group_sync�s���
�M�M�O�O�!�!��L�F�.�.�
�
����$�S�#�.B�Q�.F�*G�*G�H�H�
	,�
	,�E��K��
�
��%4�R�%8�%8�"��T�7��"�"��+�s�2��~��-�/��~��0�2�����
�C��-�-�F��d�1�7�7�6�?�?�+�+�+�F�F��
rEc�"�|j|fi|��}|�|�|��}|�|�|��}|��d�|D��}d�|D��}	g}
|D]V}|
�|jr|���n%|���������W|	D]d}t
j||j�	d����}|D]2}
|
�|jr|
���n|
���3�e|j
|
�}g}|D]g}t||���}|�d��r*ttj|j�����|d<|�|���h|S)z:
        :return tuple: (max count, list of dict)
        Nc�(�g|]}|jdk�
|��S�r��r��rZ�orders  rCr�z IPList.fetch.<locals>.<listcomp>�s,������u�/@�I�/M�/M��/M�/M�/MrEc�(�g|]}|jdk�
|��Sr*r+r,s  rCr�z IPList.fetch.<locals>.<listcomp>�s,������u�/@�I�/M�/M��/M�/M�/MrE�.)�excluder)r)r�offset�limitr"�desc�
list_priorityr)�	get_nodesr��splitrr%rfr-r)r�r�r1r2r�exclude_fieldsrr��
purpose_order�others_order�ordersr-�nodes�node�rows�row�entrys                 rC�fetchzIPList.fetch�s���
�C��Y�6�6�+�6�6�������� � �A���������A�����#+����M���#+����L��F�'�
�
���
�
��z�4�C�%�%�'�'�'��*�*�,�,�1�1�3�3�����
&�
G�
G��$�.���*�0�0��5�5����"�G�G�D��M�M���"E�$�)�)�+�+�+��F�F�F�F�G���
�F�#�A����	�	�C�!�#�~�>�>�>�E��y�y��#�#�
N�#0�����1L�1L�1L�#M�#M��i� ��K�K�������rEc�v�|s
Jd���	t|jdi|��|��S#t$r|cYSwxYw)z;Return matching row's field value or `default` if not foundzprovides kwargs to find by themNrY)�getattrrfr)r��fieldr�r\s    rC�	get_fieldzIPList.get_field�sc���8�8�8�8�8�v�	��7�3�7�,�,�V�,�,�e�4�4�4���	�	�	��N�N�N�	���s�)�8�8c�j�|�|j|j���|���|j|kz��}|�>|j|k}|s||j���z}|�|��}|�|�|j|k��}|SrK)	r�rOror�rpr�r��is_nullr?)r�r�r�r?r�r&s      rC�fetch_non_expired_queryzIPList.fetch_non_expired_query�s����J�J�s�v�s�~�.�.�4�4��n�n���
�3�<�8�#;�<�
�
���"��_��3�F��
4��#�/�1�1�3�3�3��������A��������w�.�/�/�A��rEc#�K�|�|||��}	|������Ed{V��dS#t$rYdSwxYwrK)rG�dicts�iterator�RuntimeError)r�r�r�r?r�s     rC�fetch_non_expiredzIPList.fetch_non_expired�sv�����'�'��+�w�G�G��	��w�w�y�y�)�)�+�+�+�+�+�+�+�+�+�+�+���	�	�	��F�F�	���s�,A	�	
A�Ac#�K�|�tjtjtj��}|�tj|kt���z��}|�|�|j|k��}|�	��D]'}t|d|d|d��V��(dS)zh
        Fetch listname the most efficient (though experiementally found)
        way possible.
        Nr^r>r?)r�rkr^r>r?r�r�rpr�rIr3)r�r�r�r�r>s     rC�fetch_ipnetwork_listzIPList.fetch_ipnetwork_list�s�����
�J�J�v�-�v�~�v�~�N�N��
�G�G�V�_��0�V�5F�5F�5H�5H�4H�H�I�I��������
�f�,�-�-�A��7�7�9�9�	�	�C�#��%�&��I���I����
�
�
�
�	�	rEc�f�	t�|���jS#t$rYdSwxYw)Nr�)rkrfr�r)r�rOs  rC�get_listnamezIPList.get_listname
s@��	��:�:��:�$�$�-�-���	�	�	��D�D�	���s�"�
0�0c�0�|jdk|jdz	zS)�*The result has to be passed to .where(...)rNrn�r�s rC�is_expirablezIPList.is_expirables ����!�#���4�)?�'@�@�@rEc���|tjkr|���S|���|jt	|ptj����kzS)rR)rkrlrTror�rm)r�r�s  rCrpzIPList.is_expiredsY���F�L�(�(��#�#�%�%�%����!�!��N�c�-�">�4�9�;�;�?�?�?�
�	
rEc�b�|tjko|jtjkp
|j|kS)z@Whether the ip record lives longer than given *expiration* time.)rkrlro�r�ros  rC�lives_longerzIPList.lives_longers.���V�\�)�
��O�v�|�+�K�t���/K�	
rEc��|tjkoJ|rG|�dtj��tjkp|�d��|kndS)zZWhether the properties for ip lives longer than given *expiration*
        time.

        roT)rkrlrf)r�r�ros   rC�lives_longer_propzIPList.lives_longer_prop%sX���V�\�)�
��
�J�N�N�<���6�6�&�,�F�
9��~�~�l�+�+�j�8���		
rEc�b�|jtjko|tjkp
|j|kS)z>Whether the ip record lives less than given *expiration* time.)rorkrlrWs  rC�
lives_lesszIPList.lives_less2s.����&�,�.�
��&�,�&�F�$�/�J�*F�	
rEc��|j}t|||��}|j|jko|j|jko|j|jkS)z(Analog of 3.7+ self.ip_network.subnet_of)rMr3r?r^�broadcast_address)r�rArBr?�a�bs      rC�	subnet_ofzIPList.subnet_of8sQ���O���c�4��1�1��
�I���"�
;��!�Q�%6�6�
;��#�q�':�:�	
rEc�d�|�
Jd���|�
Jd���t|jpd|��|_t|jpd|��|_|j|k}||_|�||_|�|���|jj�||���|j|jd�S)zT
        Update blocking properties

        :return tuple: real expiration
        Nz'expiration' must not be Nonez'deep' must not be Noner)�force_insertr�)ror)	�maxrorr�r��saver�r�r�)r�rorr�r��primary_key_changeds      rC�update_propertieszIPList.update_propertiesBs����%�%�'F�%�%�%����!:�����d�o�2��J�?�?�����	��Q��-�-��	�"�m�x�7�� ��
��� �D�K�
	
�	�	�2�	�3�3�3����!�!�(�t�!�4�4�4��/��I�
�
�	
rEc���t|��\}}}|�|j|����d�����|���|j|k|j|k|j	|k���
td��������
d��}|D]
}|jcSdS)zReturn the name of highest priority list that contains the *ip*.
        Return None if *ip* not in any list or record expired.r�r8N)r2r�r�r4rr�rpr^r>r?rrr3r2)r�rOrArBr?r��rs       rC�effective_listzIPList.effective_list_s���
-�R�0�0���T�7��J�J����!�!�#�#�)�)�*�5�5�
�
��U����!�!�!��#�s�*���t�#���w�&�	���X�c�*�o�o�*�*�,�,�
-�
-�
�U�1�X�X�	
��	�	�A��:�����trEc��|jj���5|�|||��cddd��S#1swxYwYdS)z.Update ip lists on CaptchaDosAlert atomically.N)�_metar��atomic�*_blacklist_graylisted_on_captcha_dos_alert)r�rOrors    rC�)blacklist_graylisted_on_captcha_dos_alertz0IPList.blacklist_graylisted_on_captcha_dos_alertws���
�Y�
�
&�
&�
(�
(�	�	��A�A��J����	�	�	�	�	�	�	�	�	�	�	�	����	�	�	�	�	�	s�A�A�
Ac���t���\}}}g}t��d���}|D�]H}	|	jtjkr;|	�t
j����rtd��d���cS|	jtjkrB|	�t
j����r|	j	rtd��d���cS|	jtjkrH|	�|��s|	�
|��st��d|	j�d|����cS|	j�kr6|	�|��s!|�
|	j|	jf����J|D] \}
}|�|
|g��}|sJ��!t j�|��|kt j|kt j|kf}
t!�t j��j|
����}|d�|D��z
}t!���j|
����|r&t2�d	�d
�|D����|��tj||d���t2�d
���t9�tjft9|���i�fd�|D�����S)a�Update ip lists on CaptchaDosAlert.

        Spec [1]:
        if search(ip, "WHITE"):  # should not really happen
            return

        existing_black_supernets = search(ip, "BLACK")
        if any(n.expiration >= expiration for n in existing_black_supernets):
            # should not really happen
            return

        existing = search_exactly(ip, "BLACK")
        if existing.manual:
            # Do nothing if already added manually
            return
        else:
            # exact match with less expiration, remove it
            # and replace with new expiration later
            remove(ip, "BLACK")

        # it can really exist only in GRAY list
        for listname in ["GRAY", "GRAY_SPLASHSCREEN", "IGNORE"]:
            existing = search_exactly(ip, listname)
            if existing and existing.expiration <= expiration:
                remove(ip, listname)

        add(ip, "BLACK", expiration)

        [1]: https://gerrit.cloudlinux.com/#/c/61260/14/src/handbook/message_processing/local_captcha_dos.py

        :param ip: attackers ip
        :param expiration: when record will expired
        :return: Union[Dict, Exception]
        r8rnz!Don't blacklist whitelisted ips [z] on CaptchaDosAlertz*Don't blacklist manually blacklisted ips [z. is already blacklisted for long enough time: z >= c�4�g|]}|jtjf��SrY)rOrkr~)rZris  rCr�zEIPList._blacklist_graylisted_on_captcha_dos_alert.<locals>.<listcomp>�s!��F�F�F�!���v�}�-�F�F�FrEzRemoved %s from %s listsc��g|]\}}|��SrYrY)rZ�_�Ls   rCr�zEIPList._blacklist_graylisted_on_captcha_dos_alert.<locals>.<listcomp>�s��0K�0K�0K�t�q�!��0K�0K�0KrEF)rOr�rorr�zPut %s on the BLACK listc���g|]	\}}�|f��
SrYrY)rZrsr�rOs   �rCr�zEIPList._blacklist_graylisted_on_captcha_dos_alert.<locals>.<listcomp>�s!���G�G�G�K�A�x�"�h��G�G�GrE)�	blocklist�unblocklist)r2rk�find_closest_ip_netsr�r�rXrmr&r�r�r\rorMr"rOr��
IgnoreListr^�bin_andr>r?r�r�r�r��logger�infor��dict)r�rOror�networkrBr?rw�	supernetsrA�ip_r�r��	is_subnet�ignore_subnetss `             rCrnz1IPList._blacklist_graylisted_on_captcha_dos_alert�s[���L"1��!4�!4����w����/�/��q�/�A�A�	��$	;�$	;�C��|�v�|�+�+��0@�0@�����0M�0M�+��v��r�r�������
����,�,��$�$�T�Y�[�[�1�1�-��J�-��v��r�r�������
�|�v�|�+�+�� � ��,�,�,�47�N�N�:�4N�4N�,�
��������"�
�	�������~��#�#�C�,<�,<�Z�,H�,H�#��"�"�C�F�C�L�#9�:�:�:��)�	 �	 �M�C���/�/��h�Z�@�@�L����<��
�&�.�.�t�4�4��?���$�&���'�)�
�	�
���j�m�,�,�2�I�>�F�F�H�H�	�	�F�F�~�F�F�F�F�������!�9�-�5�5�7�7�7��	��K�K�*�B�0K�0K�{�0K�0K�0K�
�
�
�
	�
�
���\�!���	�	
�	
�	
�	���.��3�3�3���V�\�"�D�J�$?�$?�$?��H�G�G�G�;�G�G�G�
�
�
�	
rEc�B�t|j|j|j��SrK�r3r^r>r?r�s rCrMzIPList.ip_network�s"�� �� �$�,���
�
�	
rE)ror�r2c�,�t|��\}}}|����t||||f����}	|	�|�|����}	|�-|	�|j�|����}	|	�|j�	����}	|�|	�|j
|k��}	|�|	�|��}	t|	��S)a�
        Returns all supernets containing given network (*ip*)
        that are not expired by *expiration* time.

        :param ip: ip network to lookup
        :param listname: list of listnames
        :param limit: number of supernets to return
        :param expiration: seconds since the epoch or None
           None means "use the current time"
        :return: list of matching supernets, ordered by netmask
                 (desc: from smallest to largest)
        )
r2r�r�rIrpr�r�rr>r3r�r2r)
r�rOr�r2ror�rArBr?r�s
          rCrxzIPList.find_closest_ip_nets�s���,-�R�0�0���T�7��J�J�L�L���$�S�3��g�*>�?�?�
�
��
�G�G�S�^�^�J�/�/�/�0�0���������(�(��2�2�3�3�A�
�J�J�s�{�'�'�)�)�*�*��������
�f�,�-�-�A���������A��A�w�w�rEc�F��td�fd��jD����S)Nc�0��g|]\}}�j|k|f��SrYr�)rZr�r�r�s   �rCr�z(IPList.list_priority.<locals>.<listcomp>(s9���
�
�
�&�H�h����)�8�4�
�
�
rE)rr�rSs`rCr4zIPList.list_priority$sA�����
�
�
�
�*-�*@�
�
�
�
�
�	
rE�networksc#�K�t�|��t�tjtjtj���tjdktjtkztjdktjtkzzt�
|��z��}|�.|�tjtjk��}n2|�tj�
|����}|�ttjtjktj�tj��tjkztjtjkztjtjktjtjkztjtjkztjtjkzz|�dn!|���tjkz���}t)|�����D]
}t-|�V��dS)z�Yield networks which has lasting supernets with higher priority in
        db.

        If listname isn't provided ignore priorities (for unblock).

        Implemented to solve performance issue: DEF-15123
        r �NTr)�_TempIPList�fillrkr�r^r>r?r��IPV4_HOST_MASK�IPV6_HOST_MASKrpr�r�r�r
rzrorlr4r��set�tuplesr3)r�r�r�r�r�r~s      rC�filter_ips_has_supernetszIPList.filter_ips_has_supernets.s����	����"�"�"��M�M��'�����
�
��%��.�A�%�&�.�N�*J�K��N�a�'�F�N�n�,L�M�O��!�!�-�0�0�0�	
2�
�
�		
��������6�<�7�8�8�A�A������+�+�I�6�6�7�7�A�
�F�F����;�#6�6��/�7�7���G�G��-�.��
�>�K�$7�7�9�$�.�&�,�>�%�0�K�4J�J�L�"�,���<�	>�#�-���=�?��$!�(��D��+�+�-�-��1E�E�'�
�
�
��8�1�8�8�:�:���	.�	.�G�#�W�-�-�-�-�-�	.�	.rEc#�pK�t�|��|�tjtjtj���t���tj	z���
ttjtjktjtjkztjtjkzt���tjkztj
tjktj
tj
kztj
tjkzztj
tjkz���}t|�����D]
}t#|�V��dS)z�Yield exact match of ips which already recorded in db with the same
        priority and later expiration (so we don't need to add them)

        Implemented to solve performance issue: DEF-15123
        rN)r�r�r�r^r>r?r�rkrpr�r
r4r�rorlr�r�r3)r�rr�r~s    rC�find_ips_with_later_expirationz%IPList.find_ips_with_later_expirationksa����	��������J�J��+��#��#�
�
�
�U�V�&�&�(�(�(�f�m�^�<�
=�
=�
�T���^�{�':�:��-��1L�L�N��~��)<�<�>��+�+�-�-��1E�E�G�$�.�&�,�>�%�0�K�4J�J�L�"�,���<�	>�
�#�-���=�?����	
�6�1�8�8�:�:���	.�	.�G�#�W�-�-�-�-�-�	.�	.rEc
#�K�t�|��|�tjtjtjtjtjtjktjtjktjtjkztjtjkzz���ttjtjktj�	tj��tjkztjtjkzt�
��tjkztjtj
ktjtjkztjtj
kzz���}|���D]\}}}}}t|||��||fV��dS)aFYield
            - subnet (include self) with less priority
                (listname and less expiration),
            - listname of the subnet
            - should_unblock which is True if subnet is exact blocked
                network with same listname

        Implemented to solve performance issue: DEF-15123
        rN)r�r�r�rkr^r>r?r�r
rzr4r�rorlr�r3)r�rr�r^r>r?r��should_unblocks        rC�"find_ip_subnets_with_less_priorityz)IPList.find_ip_subnets_with_less_priority�s�����	��������J�J��"��N��N��O��%���8� �(�F�N�:�"�2�f�6L�L�N��~��)<�<�>���


�

��$����;�#6�6��*�2�2�;�3F�G�G�"�2�3��
�>�[�%8�8�:��'�'�)�)�[�-A�A�
C� �*�f�l�:�!�,��0F�F�H�#�-���=�	?�
��
�
�	
�N�X�X�Z�Z�
		(�		(�
������#���'�����(�
(�
(�
(�
(�		(�		(rE)r�roc�*�t|��\}}}|�|j���|j|k|j|kz|j|kz��}|�|�|����}|�-|�|j�|����}|�	|j�
����}|�|�|j|k��}t|��S)z�
        Returns all lists containing given network (*ip*)
        :param ip: ip network to lookup
        :param listname: list of listnames
        :return: names of list
        )
r2r�r�r�r^r>r?rpr�rr3r�r)	r�rOr�r�rorArBr?r�s	         rC�
find_listszIPList.find_lists�s���-�R�0�0���T�7��J�J�s�|�$�$�*�*�
�
 �C�
'��{�d�"�
$��{�g�%�
'�
�
��

�G�G�S�^�^�J�/�/�/�0�0���������(�(��2�2�3�3�A�
�J�J�s�|�(�(�*�*�+�+��������
�f�,�-�-�A��A�w�w�rEc�P��t��j|it|����SrK�r�rfrh�r��queryr\r�s   �rCrfz
IPList.get��'����u�w�w�{�E�J�%A�&�%I�%I�J�J�JrEc�P��t��jdit|����S�NrY�r��
get_or_createrh�r�r\r�s  �rCr�zIPList.get_or_create��)���$�u�w�w�$�L�L�'C�F�'K�'K�L�L�LrEc�P��t��jdit|����Sr��r��
create_or_getrhr�s  �rCr�zIPList.create_or_get�r�rE)�include_itselfr��
expired_byc��	�t|��\}}�	|�|j|j|j|j���|rt|||�	f��nt|||�	f����}|�)|�|�	����}n8|tjkr(|�|�	|����}|�-|�|j�|����}|�|�|j
|k��}�	fd�|���D��S)arReturn ip_network objects containing all
        *ip* entries expired by *expired_by* from lists *listname*
        which are members of net *ip* including itself if *include_itself*

        :param ip: network to lookup members for
        :param listname: list name
        :param include_itself: whether to include ip itself as a subnet
          [default: False]
        :param expired_by: expiry date as "seconds since epoch"
           return entries expired by given *expired_by* timestamp
           - IPList.NEVER :: return all entries (regardless expiration)
           - None :: return non-expired entries

        Nc�B��g|]\}}}}t||���||f��SrY)r3)rZrArBr��er?s     �rCr�z+IPList.find_net_members.<locals>.<listcomp>"sD���
�
�
�&��T�8�Q��s�D�'�
2�
2�H�a�@�
�
�
rE)r2r�r^r>r�ror�rDrGrprkrlr�r�r�)
r�rOr�r�r�r�rArBr�r?s
         @rC�find_net_memberszIPList.find_net_members�sN���0-�R�0�0���T�7��J�J�����c�l�C�N�
�
�
�%��
M�"�3��d�G�(<�=�=�=�1�#��T�7�7K�L�L�
�
�	
���������)�)�)�*�*�A�A�
�6�<�
'�
'�������z�2�2�3�3�A��������(�(��2�2�3�3�A�������
�f�,�-�-�A�
�
�
�
�*+�(�(�*�*�
�
�
�	
rEc�T�|j�|��}|j|d�S)zQReturn list of iplists with less priority than list from given
        propertiesN��IP_LISTS�index�r�r�r�s   rC�#lists_with_less_or_equal_prioritiesz*IPList.lists_with_less_or_equal_priorities's*����"�"�8�,�,���|�E�F�F�#�#rEc�Z�|j�|��}|jd|dz�S)zRReturn list of iplists with greater priority than list from given
        listnameNr8r�r�s   rC�&lists_with_greater_or_equal_prioritiesz-IPList.lists_with_greater_or_equal_priorities0s.����"�"�8�,�,���|�K�e�a�i�K�(�(rEc�,�t|��\}}}|����|j|k|j|k|j|k|j|k��}|�|�|j|k��}|���SrK)	r2r�r�r�r^r>r?r�r�)r�rOr�r�r~rBr?r�s        rC�removez
IPList.remove:s���!0��!4�!4����w��
�
���"�"��L�H�$���7�*��K�4���K�7�"�	
�
�����K�K��
�f� 4�5�5�E��}�}���rE�to_blockc���tdt|��|j��D]L}|�||||jz����d������MdS)Nr�REPLACE)�range�len�
BATCH_SIZE�insert_many�on_conflictr�)r�r��idxs   rC�
block_manyzIPList.block_manyGsq����C��M�M�3�>�:�:�	�	�C��O�O�H�S�3���+?�%?�@�A�A�M�M��
�
��g�i�i�i�i�		�	rEc�t��t�|��td�d��jjj������}�fd��jjjD��}�j|��	tj���ttjtjktj
tj
kztjtjkztjtjkz���}g}|���D]-\}}}}	|�t%|||��|	f���.�����	|�|�������|S)zj
        Remove *ips* that are not manual from the [iplist] table.
        Return ips to unblock.
        z({})z, c3�8�K�|]}t�|��V��dSrK)rB)rZrCr�s  �rCr]z%IPList.remove_many.<locals>.<genexpr>[s>�����
�
�$)�G�C����
�
�
�
�
�
rEr)r�r�r�formatr
rlr��field_namesr�r�rkr�r?r^r>r�r�r"r3r�r�r�)
r�r�primary_key_sqlr��	to_remove�
to_unblockr^r>r?r�s
`         rC�remove_manyzIPList.remove_manyOs����	���������M�M�$�)�)�C�I�$9�$E�F�F�G�G�
�
��
�
�
�
�-0�Y�-B�-N�
�
�
��
�C�J��$�
�U�F�M�>�
"�
"�
�T���^�{�':�:��-��1L�L�N��~��)<�<�>���+�*>�>�@����	��
�;D�;K�;K�;M�;M�	�	�7�O�W�g�x����%�o�w��H�H���
�
�
�
�	�
�
�����?�.�.�y�9�9�:�:�B�B�D�D�D��rE)rFrK�NN)NNNNNN)srwrxryrzr�r�r�r�r�r�r�	enumerate�reversedr�rr!r~�IPv4�VERSION_IP4�IPv6�VERSION_IP6r�r�rrOrr�r
r�rlr rorrrrrrr�r�r�r	r^r>r?rr�r�r�rr�r�r�r}r�r�r�r�rr�r1r�boolr�r�rrrr�r�r�r�rrr'�	frozensetr@rDrGrLr
rNrPrTrprXrZr\rargrjrorn�propertyrMrLrxr4rrr�r�r�r�rfr�r�r�rr�r�r�r�r��
__classcell__�r�s@rCrkrk�sb�������;�;� �K�
�E�
�E��D�+���u�d�$5�6�H���i�i����(:�(:�;�;�<�<���K��K�
�F��K��K��J���������&
���	�	�	�B��y�
��U�/�6�6�u�z�z�(�7K�7K�L�L�M�M�N����H�
�E��������J�
�I�4�(�(�(�M��L�
�3�3�
�
�
�E�
�<�T�"�"�"�D��i�T�"�"�"�G��i�T�|�<�<�<�G�"�\�u�e�<�<�<�N��\�u�d�
3�
3�
3�F��,�D�)�)�)�K�#�|��u�=�=�=��#�l��.�.�.�O��l��&�&�&�G��l��&�&�&�G�
�I�
��E�E�K�K�K����E�F�F�
�
�
�
�E���������L�L�L�L�L�L�L�L�
�(�(�3�-�(�C�(�(�(��\�(���s��s�����\���	
�h�t�n�	
�	
�	
��\�	
��	
����	
�3�	
�	
�	
��\�	
��!�E�#�s�(�O�!�!�!��[�!���!�
;�;��;��#�Y�;��	;�
�;��
;�;�;��[�;�z������[���
�	���#�{�K�/�0���s�)���	����[��:����[������[��B�������
�����[��&�������?�?�?��[�?�B�B�D��I�B�B�B��[�B���t�I������[��,���� �y�{�{�
5�5���9�5�5�5��[�5�n�����[�������[�������[����D���	�)�	�����[�� ����[���A�A��[�A��
�
�
��[�
�
�
�
��

�

��[�

�
�
�
�
�
�
�
�
�
�
�:��	��h�s�m�����[��.����[���u
�u
��[�u
�n�
�
��X�
�
�#��	"���"�"�"��)�'��)>�>�?�"��s�)�"��	"�"�"��[�"�H�
�
��[�
��%)��	:.�:.��y�$�s�C�x�.�0�1�:.��D�>�:.�:.�:.��[�:.�x�&.��y�$�s�C�x�.�0�1�&.�&.�&.��[�&.�P�8(��y�$�s�C�x�.�0�1�8(�8(�8(��[�8(�t���
������s�)�����[��:�K�K�K�K��[�K��M�M�M�M��[�M��M�M�M�M��[�M��#�+
�
���+
�+
�+
��+
��s�)�+
��
+
�
�e�I�s�C�'�(�	)�+
�+
�+
��[�+
�Z�$��$�	�#��$�$�$��[�$��)��)�	�#��)�)�)��[�)��
�
�	�
�S�
�
�
��[�
���$�t�*�����[���$��5��D��1�2�$�	
�e�I�s�N�#�	$�$�$�$��[�$�$�$�$�$rErkc��eZdZed���Zed���Zed���Zed���Zed���Z	edd���Z
Gd�d��Zed���Z
ed	���Zed
eeeffd���ZdS)
r�Fr�rTr�c�<�eZdZejZdZedddd��ZdS)�_TempIPList.Meta�
tmp_iplistr^r>r?r�N)	rwrxryr(r�r�r�rr�rYrErCr�r��s4�������;����"�l��y�)�Z�
�
���rEr�c�D�|jj�d��dS)Na�
            CREATE TEMPORARY TABLE IF NOT EXISTS tmp_iplist
            (
                network_address INT, netmask INT, version INT,
                listname TXT VARCHAR(255) NOT NULL CHECK
                (listname in ('WHITE','BLACK','GRAY','GRAY_SPLASHSCREEN')),
                priority INT,
                expiration INT,
            PRIMARY KEY (network_address, netmask, version, listname))
        )rlr��execute_sqlrSs rC�_createz_TempIPList._create�s.���	��&�&�	
�	
�	
�	
�	
�	
rEc�R�|������dSrK)r�r�rSs rC�_clearz_TempIPList._clear�s"���
�
���������rErc�$�t|d��r|���n|}tdt��5|���|���d�|D��}ddd��n#1swxYwY|r�tdt��5d}t
dt|��|��D]4}|�||||z����	���5	ddd��dS#1swxYwYdSdS)N�itemszprepare tmp_iplist to fillc���g|]g\}}ttgd�gt|���t�|���t�|���������hS))r^r>r?r�r�ro)r}�zipr2rkr�r�)rZrO�props   rCr�z$_TempIPList.fill.<locals>.<listcomp>�s������$�B��#������,�R�0�0��#�C�C�D�I�I��#�6�6�t�<�<��������rEzfill tmp_iplist�r)
�hasattrr�r,r{r�r�r�r�r�r�)r�r�data�
batch_sizer�s     rCr�z_TempIPList.fill�s���$�S�'�2�2�;�c�i�i�k�k�k���
�0�&�
9�
9�	�	��K�K�M�M�M��J�J�L�L�L���$!$�%���D�	�	�	�	�	�	�	�	�	�	�	����	�	�	�	�0�	L��)�6�2�2�
L�
L� �
� ��C��I�I�z�:�:�L�L�C��O�O�D��s�Z�/?�)?�$@�A�A�I�I�K�K�K�K�L�
L�
L�
L�
L�
L�
L�
L�
L�
L�
L�
L�
L����
L�
L�
L�
L�
L�
L�	L�	Ls$�5A=�=B�B�AD�D�
DN)rwrxryr r^r>r?rr�r�ror�r�r�r�rrr
r�rYrErCr�r�ws������"�l��.�.�.�O��l��&�&�&�G��l��&�&�&�G��y�e�$�$�$�H��|��'�'�'�H��������J�
�
�
�
�
�
�
�
��
�
��[�
�����[���L�u�T�8�^�,�L�L�L��[�L�L�LrEr�c��eZdZdZd\ZZedd���Zedd���Z	Gd�d	��Z
ed
���Zed���Z
dS)
�LastSynclistzFUsed to track how up-to-date are lists synced from Correlation server.)rO�hashTrr�F)r�r�c�$�eZdZejZdZdZdS)�LastSynclist.Meta�
last_synclistr�N)rwrxryr(r�r�r�r�rYrErCr�r��s�������;��"�����rEr�c��|�tj������|j|k�����dS)N)�	timestamp)rbrmr��namer�)r�r�s  rC�update_timestampzLastSynclist.update_timestamp�sC���
�
�T�Y�[�[�
�)�)�/�/���D�0@�A�A�I�I�K�K�K�K�KrEc�H�|�|ddi���\}}|jS)Nr�r)r��defaults)r�r�)r�r��objrss    rC�
get_timestampzLastSynclist.get_timestamp�s+���"�"���Q�7G�"�H�H���Q��}�rEN)rwrxryrzr5�HASHrr�rr�r�r�r�r�rYrErCr�r��s�������P�P��H�B���
��a�0�0�0�I��9�%�T�2�2�2�D���������
�L�L��[�L�����[���rEr�c��eZdZdZGd�d��Ze��Zed���Ze	d���Z
e	d���ZdS)	�WhitelistedCrawlerz^
    Crawlers for which local alerts must not add IP
    to the :attr:`IPList.GRAY` list.
    c� �eZdZejZdZdS)�WhitelistedCrawler.Meta�whitelisted_crawlersN�rwrxryr(r�r�r�rYrErCr�r��s�������;��)���rEr�Fr�c��tj���5|�|������}|D]}t
�||����	ddd��dS#1swxYwYdS)N)�description)�
crawler_id�domain)r(r�rm�insertr��WhitelistedCrawlerDomainr�)r�r��domains�inserted_id�ds     rC�addzWhitelistedCrawler.add�s���
�[�
�
�
!�
!�	�	��*�*��*�=�=�E�E�G�G�K��
�
��(�/�/�*�1�0�����
�	�	�	�	�	�	�	�	�	�	�	�	����	�	�	�	�	�	s�A
A7�7A;�>A;c��|����|j���|���|��}t
���}t
||��}g}|�d���}|D]6}|j|jd�|j	D��d�}	|�
|	���7||fS)NT)�clear_limitc��g|]	}|j��
SrY)r�)rZrs  rCr�z,WhitelistedCrawler.fetch.<locals>.<listcomp>�s��>�>�>��A�H�>�>�>rE)rr�r)r�rr�r2r1rr$rrrr")
r�r2r1�crawlers_query�
domains_query�crawlers_with_domains_queryr#�	max_count�crawler�items
          rCr@zWhitelistedCrawler.fetch�s���
�J�J�L�L�!�!�#�/�2�2�8�8��?�?�F�F�v�N�N�	�1�7�7�9�9�
�&.�~�}�&M�&M�#���"�(�(�T�(�:�:�	�2�	 �	 �G��j�&�2�>�>�g�o�>�>�>���D�

�M�M�$������&� � rEN)rwrxryrzr�r!rr"r�r�rr@rYrErCr�r��s���������
*�*�*�*�*�*�*�*�
��	�	�B��)��'�'�'�K�����[���!�!��[�!�!�!rEr�c�v�eZdZdZGd�d��Ze��Zeeddd���Z	e
d���Zd	S)
rzBDomain names used to check if IP is a :class:`WhitelistedCrawler`.c� �eZdZejZdZdS)�WhitelistedCrawlerDomain.Meta�whitelisted_crawler_domainsNr�rYrErCr�rs�������;��0���rEr�F�CASCADEr�r��	on_delete�related_namer�N)rwrxryrzr�r!rrr�r
r"r�rYrErCrrs�������L�L�1�1�1�1�1�1�1�1�
��	�	�B��o��
���	���G��Y�E�
"�
"�
"�F�F�FrErc	���eZdZdZdZdZed���Zeded�	ee����g���Z
edd�	��ZGd
�d��Z
eded
ededefd���ZdS)�RemoteProxyGroupz9Groups multiple remote proxies together with common data.r��
imunify360Fr�zsource in ('{}', '{}')r�Tr�c�$�eZdZejZdZdZdS)�RemoteProxyGroup.Meta�remote_proxy_group))�r��sourceTN)rwrxryr(r�r�r��indexesrYrErCr�r's�������;��'��/���rEr�r�r�enabledric��t�||���}|j|krdS||_|���dS)z�Set group's enabled status.

        Group is identified by name and source. Returns True if enabled
        status has changed, False otherwise (it was a noop).rFT)rrfr re)r�r�rr r�s     rC�set_enabledzRemoteProxyGroup.set_enabled,sG��!�$�$�$�v�$�>�>���=�G�#�#��5���
�
�
�
�����trEN)rwrxryrz�MANUAL�
IMUNIFY360rr�rr�rrr r�r�r�r�r"rYrErCrrs�������C�C��F��J��9�%� � � �D��Y�
��E�*�1�1�&�*�E�E�F�F�
����F��l��t�4�4�4�G�0�0�0�0�0�0�0�0�
�
�s�
�C�
�$�
�4�
�
�
��[�
�
�
rErc
��eZdZdZeed���Zed���ZGd�d��Z	e
deedeedee
d	eefd
���Ze
deded
eefd���Ze
ded
eefd���ZdS)�RemoteProxyz!Remote Proxy networks in a group.Fr�c� �eZdZejZdZdS)�RemoteProxy.Meta�remote_proxyNr�rYrErCr�r(As�������;��!���rEr��by_group�	by_sourcer ric��|�tjtjtj|j���t��}|�#|�tj|k��}|�#|�tj|k��}|�#|�tj|k��}t|�	tj���
����S)z�Returns a list of remote proxy networks as dicts.

        Results are optionally filtered by group name, source, and enabled
        status.)r�rrr�r r~r
r�rrrI)r�r*r+r r�s     rCrzRemoteProxy.listEs���
�J�J��#��!��$��K�	
�
�
�$��
 �
 �	
������(�-��9�:�:�A�� ����(�/�9�<�=�=�A������(�0�G�;�<�<�A��A�J�J�/�4�5�5�;�;�=�=�>�>�>rEr�rr�c���t�||���\}}|D]R}tjt	j|����}t
||j���}|����SdS)z>Adds networks to a list of remote proxy in group name, source.r)r~�group_idN)	rr�r5r`rLrMr&rre)r�r�rr�r�rsrA�proxys        rC�add_manyzRemoteProxy.add_many^sw��$�1�1�t�F�1�K�K���q��	�	�C��%�i�&:�3�&?�&?�@�@�C���e�h�?�?�?�E��J�J�L�L�L�L�		�	rEc���d�|D��}g}t����t���tj|z���tj|k��}t|��D]0}|�|j��|�	���1t�t���ttj���t���
tjtj��dk��}t|��D]}|�	���|S)zrDeletes networks from remote proxy lists.

        Only networks coming from groups with given source are deleted.c�Z�g|](}tjtj|������)SrY)r5r`rLrM)rZrAs  rCr�z/RemoteProxy.delete_networks.<locals>.<listcomp>ns;��
�
�
�?B�B��	� 4�S� 9� 9�:�:�
�
�
rEr)r&r�r
rr�r~rrr"�delete_instancerrrrr#�COUNTr)r�rr�r�r�r/r�s       rC�delete_networkszRemoteProxy.delete_networkshs=��
�
�FN�
�
�
����
��� � �
�T�"�
#�
#�
�U�;�&�(�2�
3�
3�
�U�#�*�f�4�
5�
5�		
��!�W�W�	$�	$�E��N�N�5�=�)�)�)��!�!�#�#�#�#�
�#�#�$4�5�5�
�T�+�t��
/�
/�
�X�&�
'�
'�
�V�B�H�[�^�,�,��1�
2�
2�		
��!�W�W�	$�	$�E��!�!�#�#�#�#��rEN)rwrxryrzrrr�r"r~r�r�rr�r�rr}rr0r5rYrErCr&r&:s*������+�+��O�,�5�9�9�9�E��i�U�#�#�#�G�"�"�"�"�"�"�"�"��?��3�-�?��C�=�?��$��	?�

�d��?�?�?��[�?�0��C�����S�	�����[����S��D��I�����[���rEr&c�N��eZdZdZed���Zed���Zed���Zed���Z	Gd�d��Z
e�fd���Ze�fd���Z
e�fd���Ze�fd	���Zed
edeefd���Zed
eefd���Z�xZS)ryz�
    IP addresses from this list are not blocked by firewall. However,
    they still can be placed to other lists by either server or local
    events or by user request.
    Fr�c�>�eZdZejZdZeddd��ZdZ	dS)�IgnoreList.Meta�ignore_listr^r>r?r�Nr�rYrErCr�r8�s3�������;�� ��"�l�#4�i��K�K�����rEr�c�P��t��jdit|����S)r�rY�r�r�rcr�s  �rCr�zIgnoreList.create�s*����u�w�w�~�9�9� 0�� 8� 8�9�9�9rEc�P��t��jdit|����Sr�r�r�s  �rCr�zIgnoreList.create_or_get�r�rEc�P��t��j|it|����SrKr�r�s   �rCrfzIgnoreList.get�r�rEc�P��t��jdit|����Sr�r�r�s  �rCr�zIgnoreList.get_or_create�r�rE�supernetric#�,K�t|��\}}}|����|j�|��|k|j|k|j|k��}|D]$}t|j|j|j��V��%dSrK)r2r�r�r^rzr>r?r3)r�r?�addressrBr?r�r>s       rC�subnetszIgnoreList.subnets�s�����!0��!:�!:����w��J�J�L�L���
�
 �
(�
(��
.�
.�7�:��K�4���K�7�"�
�
��
�	�	�C�#��#�S�[�#�+���
�
�
�
�	�	rE�	to_deletec��t|��}|D]k}t|��\}}}|����|j|k|j|k|j|k������ldSrK)r�r2r�r�r^r>r?r�)r�rC�uniquerrArBr?s       rCr�zIgnoreList.remove�s����Y�����	�	�D�%4�T�%:�%:�"�G�T�7��J�J�L�L����#�w�.���t�#���w�&�
�
��g�i�i�i�i�
	�	rE)rwrxryrzrrOr r^r>r?r�r�r�r�rfr�r1r
rBrr�r�r�s@rCryry�s����������
���	�	�	�B�"�l��.�.�.�O��l��&�&�&�G��l��&�&�&�G����������:�:�:�:��[�:��M�M�M�M��[�M��K�K�K�K��[�K��M�M�M�M��[�M��
�y�
�X�i�-@�
�
�
��[�
���t�I������[�����rEryc���eZdZdZed���Zedede�de	�de
�d���g���Zed���ZGd	�d
��Z
e	dd���Zed
���Zedd���ZdS)�BlockedPortz]Port blocking configuration.

    Effective when `FIREWALL.port_blocking_mode == ALLOW`.
    Fr�zproto in ('z', 'r�r�Tc�(�eZdZejZdZdZdZdS)�BlockedPort.Meta�blocked_portr�)))�port�protoTN�	rwrxryr(r�r�r�r�rrYrErCr�rI�s&�������;��!����
���rEr�Nc�B�|�M|�|j�|��tj�|��z��}|�(|�t	t|����}|�#|�t
j|k��}|SrK)r�rrP�
IgnoredByPortrTr-r)r�r�rrrs     rC�_add_filter_argszBlockedPort._add_filter_args�s����!������$�$�Z�0�0��'�0�0��<�<�=���A������,�]�E�B�B�C�C�A��&�������7�8�8�A��rEc�^�|�|j������tt
j���tt
jtjtjk���}|j	|fi|��}|�
��S)Nr)r�r�distinctr
rOrrr-rrPr)r�rr�s   rCrzBlockedPort.fetch_count�s���
�J�J�s�v���
�X�Z�Z�
�T�-���
1�
1�
�T����!�)�W�Z�7����		
�
!�C� ��2�2�k�2�2���w�w�y�y�rEr�rc
��|�|j|j|j|jt
jt
j�d�����t
tj
���ttj
t
jtjk���}|j
|fi|��}|�|j|jt
j��}d�}g}tj|���|���D]+\}}d�|D��}	|	|d<|�|���,||||z�S)N�
ip_commentrc� ���fd�dD��S)Nc�"��i|]}|�|��SrYrY)rZ�keyr>s  �rC�
<dictcomp>z8BlockedPort.fetch.<locals>.group_key.<locals>.<dictcomp>s,������"%��S��X���rE)rrKrLrrY�r>s`rC�	group_keyz$BlockedPort.fetch.<locals>.group_keys.�������)K����
rE)rWc�@�g|]}|d�
|d|dd���S)rONrT)rOrrY)rZrOs  rCr�z%BlockedPort.fetch.<locals>.<listcomp>"s=�������d�8�'��$�x�B�|�,<�=�=�'�'�'rEr)r�rrKrLrrOrOrr
rrr-rrPr�	itertools�groupbyrIr")
r�r2r1rr�rZr#�
port_protor�ignored_ipss
          rCr@zBlockedPort.fetchsP��
�J�J������	���� ��%�+�+�L�9�9�

�
��T�-���
1�
1�
�T����!�)�W�Z�7����	
�"
!�C� ��2�2�k�2�2��
�J�J�s�x���M�,<�=�=��	�	�	�
��(�0������	�J�J�J�	&�	&�O�J��������K�
!,�J�u���M�M�*�%�%�%�%��f�v��~�-�.�.rE)NNN)r�r)rwrxryrzr rKrrr/r0r.rLrr�r�rPrr@rYrErCrGrG�s���������<�U�#�#�#�D��I�
��U�B��B�B�#�B�B�3�B�B�B�C�C�D�
�
�
�E�
�i�T�"�"�"�G�
�
�
�
�
�
�
�
��=A�����[������[���%/�%/�%/��[�%/�%/�%/rErGc�2��eZdZdZeeddd���Zed���Zed���Z	e
d���Ze
d���Ze
d���Z
edd�	��ZGd
�d��Ze�fd���Zeddefd���Zedd���Z�xZS)rOz7Ignored IPs for ports blocked via :class:`BlockedPort`.Frrrr�Tr�r�c�(�eZdZejZdZdZdZdS)�IgnoredByPort.Meta�ignored_by_port_protor�)))r^rOTNrMrYrErCr�rbAs&�������;��*����
���rEr�c�T��t��jdit|����dSr�r;r�s  �rCr�zIgnoredByPort.createKs.��������2�2�)�&�1�1�2�2�2�2�2rENr?c��|�|t���t��}|�|�|j|k��}|SrK)r�rGr
r�r?)r�r?r�s   rCr@zIgnoredByPort.fetchOsF���J�J�s�K�(�(�-�-�k�:�:���������w�.�/�/�A��rEc���|jj�d�t|jj��z}|jj�d|�d���|j�|���dS)Nr/z&
          CREATE TABLE IF NOT EXISTS a�ignored_by_port_proto
          (
             "id" INTEGER NOT NULL PRIMARY KEY,
             "port_proto_id" INTEGER NOT NULL,
             "ip" VARCHAR(255) NOT NULL,
             "comment" VARCHAR(255),
             "network_address" INTEGER NOT NULL,
             "netmask" INTEGER NOT NULL,
             "version" INTEGER NOT NULL,
             "country_id" VARCHAR(255),
             FOREIGN KEY ("port_proto_id")
             REFERENCES "blocked_port" ("id") ON DELETE CASCADE
        ))�safe)rlr�r�r�r��_schema�create_indexes)r�rg�optionsr�s    rC�create_tablezIgnoredByPort.create_table[s~���I�$�'�'�'�$�s�y�/?�*@�*@�@���	��&�&�


�&,�


�


�


�	
�	
�	
� 	��"�"��"�-�-�-�-�-rErK)T)rwrxryrzrrGr^rrOrr r^r>r?rr�r�r�r7r@rkr�r�s@rCrOrO-sN�������A�A� ���%�9�5����J�

���	�	�	�B��i�T�"�"�"�G�"�l��.�.�.�O��l��&�&�&�G��l��&�&�&�G��i�T�|�<�<�<�G�
�
�
�
�
�
�
�
��3�3�3�3��[�3����,�����[���.�.�.��[�.�.�.�.�.rErOc	�J��eZdZdZed���Zed���Zed���Zed���ZGd�d��Z	e
	ddeded	e
efd
���Ze
deded	efd���Ze
d���Ze
dd���Ze
dd���Ze
�fd���Z�xZS)�IPListRecordz/DB table that stores ips for given iplist_id`s.Fr�c�@�eZdZejZdZedddd��ZdZ	dS)�IPListRecord.Meta�iplistrecordr^r>r?�	iplist_id�
ipsetlistsNr�rYrErCr�rozs;�������;��!��"�l��y�)�[�
�
�����rEr��P��
ip_versionrqric#��K�|�|j|j|j���|j|k|jt|kz�����}d}|�|���	|��x}r^||�
��z
}tt|��Ed{V��|�|���	|��x}�\dSdS)z�
        Yield ips corresponding to *ip_version*, *iplist_id*.
        NOTE: It assumes exclusive access to the db until the iterator
              is exhausted.
        rN)
r�r^r>r?r�rqr7r�r2r1rrr3)r�rtrq�
chunk_sizer�r1r$s       rC�	fetch_ipszIPListRecord.fetch_ips�s����(
�I�I�b�(�"�*�b�j�A�A�
�U����*��:�!1�*�!=�=�?����V�X�X�
	����{�{�:�.�.�5�5�f�=�=�=�e�	9��e�k�k�m�m�#�F��0�%�8�8�8�8�8�8�8�8�8��{�{�:�.�.�5�5�f�=�=�=�e�	9�	9�	9�	9�	9rEc��|����|j|k|jt|kz�����S)z*How many *ip_version* ips for *iplist_id*.)r�r�rqr?r7r)r�rtrqs   rC�fetch_ips_countzIPListRecord.fetch_ips_count�sM��
�I�I�K�K�
�U����*��:�!1�*�!=�=�?����U�W�W�
	
rEc�p�|�|tjtj�d�����t|jtjk���}|�#|�tj|k��}|r#|�t||����}|S)Nrr)r��
IPListPurposer�rqrr
r�rV�r�r�rr�s    rCrzIPListRecord._fetch_query�s����J�J���!��#�)�)�$�/�/�
�
��$�}�#�-�=�3J�"J�$�
L�
L�		
������
�-��8�9�9�A��	;����-�c�5�9�9�:�:�A��rENc�V�|�||��}|���SrKrr|s    rCrzIPListRecord.fetch_count�s%�����W�e�,�,���w�w�y�y�rEc��|�||��}|�|�|��}|�|�|��}d�td�|��D��S)Nc��g|]J\}}t|j��t|j��|j|jj|jjd���KS))r^r>r?rqr�)r�r^r>r?�
iplistpurposerr�)rZr>rOs   rCr�z&IPListRecord.fetch.<locals>.<listcomp>�se��
�
�
���R�
$'�r�'9�#:�#:��r�z�?�?��;� �.�1��,�4�
�
�
�
�
rEc�F�|t|j|j|j��fSrKr�rYs rCr�z$IPListRecord.fetch.<locals>.<lambda>�s'���%��+�S�[�#�+����rE)rr1r2�map)r�r�rr1r2r�s      rCr@zIPListRecord.fetch�s�����W�e�,�,�������� � �A���������A�
�
�������
�
�
�	
rEc�T��t��jdit|����}|Sr�r;r�s   �rCr�zIPListRecord.create�s,����u�w�w�~�9�9� 0�� 8� 8�9�9���rE)rsrKr�)rwrxryrzr r^r>r?rqr�r�r6�IPListIDrr1rwr�ryrrr@r�r�r�s@rCrmrmrs��������9�9�"�l��.�.�.�O��l��&�&�&�G��l��&�&�&�G���%�(�(�(�I����������CH� 9� 9�!� 9�.6� 9�	�)�	� 9� 9� 9��[� 9�D�	
�	�	
�h�	
�3�	
�	
�	
��[�	
�����[�������[���
�
�
��[�
�2������[�����rErmc���eZdZdZed���Zed���ZGd�d��Ze	de
deedee
fd	���Ze	de
d
edefd���Zededefd
���ZdS)r{z6DB table that stores "purposes" for given iplist_id`s.Fr�c�<�eZdZejZdZedd��ZdZ	dS)�IPListPurpose.Metar�r�rqrrNr�rYrErCr�r��s0�������;��"��"�l�9�k�:�:�����rEr�rt�purposesric���ttd��|�|j������t|jtjk����tjt|k|j
�ttt|������z�������S)z=Yield all distinct iplist_id for *ip_version* and *purposes*.rr)r�r
r�rqrRr
rmr�r?r7r�r�rr�r�)r�rtr�s   rC�fetch_iplist_idszIPListPurpose.fetch_iplist_ids�s���
��q�M�M��I�I�b�l�#�#�
�X�Z�Z�
�T�,�B�L�L�4J�$J�T�
L�
L�
�U��%�)9�*�)E�E��*�.�.��c�#�x�&8�&8�!9�!9�:�:�;����V�X�X�

�

�
	
rEr�c�H�|�|j������t|jtjk����tjt|k|j|kz���	��S)z9How many iplists are there for *ip_version* and *purpose*r)
r�rqrRr
rmr�r?r7r�r)r�rtr�s   rCrzIPListPurpose.fetch_countsx��
�I�I�b�l�#�#�
�X�Z�Z�
�T�,�B�L�L�4J�$J�T�
L�
L�
�U��%�)9�*�)E�E��:��(�*����U�W�W�		
rEr�c���tjtjtjtjtjtjtjtji|S)z,Return purpose corresponding to iplist name.)	rkr�r�r�r{r�r|r�r}r�s rC�listname2purposezIPListPurpose.listname2purposes>��
�L�'�-��L�'�,��K����$�g�&:�	
�
��	rEN)rwrxryrzrr�r rqr�r�r6r
r�r�r�r�rr�r�r�rYrErCr{r{�s������@�@��i�U�#�#�#�G���%�(�(�(�I����������
�!�
�-5�g�->�
�	�(�	�
�
�
��[�
� �
�I�
��
�C�
�
�
��[�
���3��7�����\���rEr{)mrzrLr\�loggingrm�datetimer�enumr�	functoolsrrrr�operatorr	r
�typingrrr
rrrrrrr�blinkerr�peeweerrrrrrrrrrr r!r"r#r$�playhouse.shortcutsr%�"defence360agent.contracts.messagesr&�defence360agent.modelr'r(�$defence360agent.model.simplificationr)�defence360agent.utilsr*r+r,�im360.model.countryr-�im360.utils.netr.r/r0r1r2r3r4�im360.utils.validater5r6r7r�r��	getLoggerrwr{r�r��V4r�r��V6r�rDrGrIrTrVrcrhr�rprrr�r�rkr�r�r�rrr&ryrGrOrmr{rYrErC�<module>r�s���2�2�����������������������������������.�.�.�.�.�.�.�.�������$�$�$�$�$�$�$�$�����������������������������������������������������������������".�-�-�-�-�-�5�5�5�5�5�5�1�1�1�1�1�1�1�1�=�=�=�=�=�=�O�O�O�O�O�O�O�O�O�O�'�'�'�'�'�'�������������������A�@�@�@�@�@�@�@�@�@���	��	�8�	$�	$�� ����\�!:�!:�;�;�A�>�� ����W�!5�!5�6�6�q�9������$������$�����;��',�S�#�s�]�';�����$���;��',�S�#�s�]�';�����$���;��',�S�#�s�]�';�����$���&F�F�F����"���D�d�D�D�D�D�

�
�
�
�
��
�
�
������c�4����@N�N�N�N�N�U�N�N�N�b$DL�DL�DL�DL�DL�%�DL�DL�DL�N�����5����0'!�'!�'!�'!�'!��'!�'!�'!�T#�#�#�#�#�u�#�#�#�&$�$�$�$�$�u�$�$�$�NK�K�K�K�K�%�K�K�K�\A�A�A�A�A��A�A�A�H^/�^/�^/�^/�^/�%�^/�^/�^/�BB.�B.�B.�B.�B.�E�B.�B.�B.�Jp�p�p�p�p�5�p�p�p�f4�4�4�4�4�E�4�4�4�4�4rE