SYSTEM AND METHOD FOR RECONFIGURABLE INTELLIGENT SURFACE (RIS)-ASSISTED ENERGY-EFFICIENT (EE) RADIO ACCESS NETWORK (RAN) USING HIERARCHICAL REINFORCEMENT LEARNING

Abstract

A method, system and apparatus for reconfigurable intelligent surface-assisted energy-efficient radio access networks using hierarchical reinforcement learning are disclosed. A method in a network node operating as a meta-controller and configured to communicate with a wireless device and a plurality of sub-controllers is provided. The method includes determining a state of the meta-controller including traffic load ratios of the plurality of sub-controllers. The method also includes receiving an extrinsic reward that is based on an energy efficiency of a cell including the plurality of sub-controllers. The method further includes selecting a goal according to a policy, the selected goal being an on/off state of each of the plurality of sub-controllers, the policy being selected to increase the extrinsic reward. The method includes configuring the plurality of controllers with the selected goal and an indication of the policy for selecting the goal.

Claims

1. A method implemented in a network node operating as a meta-controller and configured to communicate with a wireless device, WD, and a plurality of sub-controllers, the method comprising: determining a state of the meta-controller, the state of the meta-controller including traffic load ratios of the plurality of sub-controllers; receiving an extrinsic reward, the extrinsic reward being based at least in part on an energy efficiency of a cell including the plurality of sub-controllers; selecting a goal according to a policy, the selected goal being an on/off state of each of the plurality of sub-controllers, the policy being selected to increase the extrinsic reward; and configuring the plurality of sub-controllers with the selected goal and an indication of the policy for selecting the goal.

2. The method of claim 1, wherein the extrinsic reward is based at least in part a ratio of a sum of throughputs of network nodes in the cell to a sum of power consumptions of the network nodes in the cell.

3. The method of claim 2, wherein the extrinsic reward is further based at least in part on a penalty factor to avoid overloading and based at least in part on a number of network nodes in the cell that are overloaded.

4. The method of claim 1, wherein maximizing the extrinsic reward includes maximizing a throughput of a link between the network node and a plurality of WDs via a reconfigurable intelligent surface, RIS.

5. The method of claim 1, wherein the policy is one of a greedy policy and an e-greedy policy.

6. The method of claim 5, wherein the greedy policy provides a selected goal that increases the extrinsic reward when a random number exceeds a threshold and provides a randomly selected goal when the random number does not exceed the threshold.

7. A network node operating as a meta-controller and configured to communicate with a wireless device, WD, and a plurality of sub-controllers, the network node comprising processing circuitry configured to: determine a state of the meta-controller, the state of the meta-controller including traffic load ratios of the plurality of sub-controllers; receive an extrinsic reward, the extrinsic reward being based at least in part on an energy efficiency of a cell including the plurality of sub-controllers; select a goal according to a policy, the selected goal being an on/off state of each of the plurality of sub-controllers, the policy being selected to increase the extrinsic reward; and configure the plurality of sub-controllers with the selected goal and an indication of the policy for selecting the goal.

8. The network node of claim 7, wherein the extrinsic reward is based at least in part a ratio of a sum of throughputs of network nodes in the cell to a sum of power consumptions of the network nodes in the cell.

9. The network node of claim 8, wherein the extrinsic reward is further based at least in part on a penalty factor to avoid overloading and based at least in part on a number of network nodes in the cell that are overloaded.

10. The network node of claim 7, wherein maximizing the extrinsic reward includes maximizing a throughput of a link between the network node and a plurality of WDs via a reconfigurable intelligent surface, RIS.

11. The network node of claim 7, wherein the policy is one of a greedy policy and an e-greedy policy.

12. The network node of claim 11, wherein the greedy policy provides a selected goal that increases the extrinsic reward when a random number exceeds a threshold and provides a randomly selected goal when the random number does not exceed the threshold.

13. A method implemented in a network node operating as a sub-controller and configured to communicate with a wireless device (WD) and at least one network node operating as a meta-controller, the method comprising: receiving a goal and an indication of a policy from the meta-controller; receiving an intrinsic reward, the intrinsic reward being based at least in part on a ratio of a throughput of the sub-controller to a transmission power of the sub-controller; and selecting an action based at least in part on the goal and according to the policy, the action including adjusting the transmission power of the sub-controller to increase the intrinsic reward.

14. The method of claim 13, wherein the intrinsic reward is further based at least in part on a penalty factor to avoid overloading and based at least in part on a number of network nodes in a cell that are overloaded.

15. The method of claim 14, wherein maximizing the intrinsic reward includes maximizing a throughput of a link between the network node and a plurality of WDs via a reconfigurable intelligent surface, RIS.

16. The method of claim 13, wherein the policy is one of a greedy policy and an e-greedy policy.

17. The method of claim 16, wherein the greedy policy provides a selected action that increases the intrinsic reward when a random number exceeds a threshold and provides a randomly selected action when the random number does not exceed the threshold.

18. The method of claims 13-17 claim 13, wherein the goal is an on/off state of the sub-controller.

19.-24. (canceled)

25. The method of claim 2, wherein maximizing the extrinsic reward includes maximizing a throughput of a link between the network node and a plurality of WDs via a reconfigurable intelligent surface, RIS.

26. The method of claim 2, wherein the policy is one of a greedy policy and an -greedy policy.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0024] A more complete understanding of the present embodiments, and the attendant advantages and features thereof, will be more readily understood by reference to the following detailed description when considered in conjunction with the accompanying drawings wherein:

[0025] FIG. 1 is a schematic diagram of an example network architecture illustrating a communication system connected via an intermediate network to a host computer according to the principles in the present disclosure;

[0026] FIG. 2 is a block diagram of a host computer communicating via a network node with a wireless device over an at least partially wireless connection according to some embodiments of the present disclosure;

[0027] FIG. 3 is a flowchart illustrating example methods implemented in a communication system including a host computer, a network node and a wireless device for executing a client application at a wireless device according to some embodiments of the present disclosure;

[0028] FIG. 4 is a flowchart illustrating example methods implemented in a communication system including a host computer, a network node and a wireless device for receiving user data at a wireless device according to some embodiments of the present disclosure;

[0029] FIG. 5 is a flowchart illustrating example methods implemented in a communication system including a host computer, a network node and a wireless device for receiving user data from the wireless device at a host computer according to some embodiments of the present disclosure;

[0030] FIG. 6 is a flowchart illustrating example methods implemented in a communication system including a host computer, a network node and a wireless device for receiving user data at a host computer according to some embodiments of the present disclosure;

[0031] FIG. 7 is a flowchart of an example process in a network node for reconfigurable intelligent surface (RIS)-assisted energy-efficient (EE) radio access networks (RAN) using hierarchical reinforcement learning;

[0032] FIG. 8 is a flowchart of an example process in a wireless device for reconfigurable intelligent surface (RIS)-assisted energy-efficient (EE) radio access networks (RAN) using hierarchical reinforcement learning;

[0033] FIG. 9 is a flowchart of an example process in a network node for reconfigurable intelligent surface (RIS)-assisted energy-efficient (EE) radio access networks (RAN) using hierarchical reinforcement learning;

[0034] FIG. 10 is a flowchart of an example process in a wireless device for reconfigurable intelligent surface (RIS)-assisted energy-efficient (EE) radio access networks (RAN) using hierarchical reinforcement learning;

[0035] FIG. 11 is a block diagram of an example implementation of hierarchical reinforced learning for controlling sleep states of at least one sub-controller;

[0036] FIG. 12 is a flowchart of an example process for machine learning reinforced by feedback from the network environment; and

[0037] FIG. 13 is a block diagram of another example implementation of hierarchical reinforced learning for controlling sleep states of at least one sub-controller.

DETAILED DESCRIPTION

[0038] Before describing in detail example embodiments, it is noted that the embodiments reside primarily in combinations of apparatus components and processing steps related to reconfigurable intelligent surface (RIS)-assisted energy-efficient (EE) radio access networks (RAN) using hierarchical reinforcement learning. Accordingly, components have been represented where appropriate by conventional symbols in the drawings, showing only those specific details that are pertinent to understanding the embodiments so as not to obscure the disclosure with details that will be readily apparent to those of ordinary skill in the art having the benefit of the description herein. Like numbers refer to like elements throughout the description.

[0039] As used herein, relational terms, such as first and second, top and bottom, and the like, may be used solely to distinguish one entity or element from another entity or element without necessarily requiring or implying any physical or logical relationship or order between such entities or elements. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the concepts described herein. As used herein, the singular forms a, an and the are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms comprises. comprising. includes and/or including when used herein, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

[0040] In embodiments described herein, the joining term, in communication with and the like, may be used to indicate electrical or data communication, which may be accomplished by physical contact, induction, electromagnetic radiation, radio signaling, infrared signaling or optical signaling, for example. One having ordinary skill in the art will appreciate that multiple components may interoperate and modifications and variations are possible of achieving the electrical and data communication.

[0041] In some embodiments described herein, the term coupled, connected, and the like, may be used herein to indicate a connection, although not necessarily directly, and may include wired and/or wireless connections.

[0042] The term network node used herein can be any kind of network node comprised in a radio network which may further comprise any of base station (BS), radio base station, base transceiver station (BTS), base station controller (BSC), radio network controller (RNC), g Node B (gNB), evolved Node B (eNB or eNodeB), Node B, multi-standard radio (MSR) radio node such as MSR BS, multi-cell/multicast coordination entity (MCE), integrated access and backhaul (IAB) node, relay node, donor node controlling relay, radio access point (AP), transmission points, transmission nodes, Remote Radio Unit (RRU) Remote Radio Head (RRH), a core network node (e.g., mobile management entity (MME), self-organizing network (SON) node, a coordinating node, positioning node, MDT node, etc.), an external node (e.g., 3rd party node, a node external to the current network), nodes in distributed antenna system (DAS), a spectrum access system (SAS) node, an element management system (EMS), etc. The network node may also comprise test equipment. The term radio node used herein may be used to also denote a wireless device (WD) or a radio network node.

[0043] In some embodiments, the term macro base station (MBS) refers to a network node that is configured to operate as a meta-controller. The term small base station (SBS) refers to a network node that is configured to operate as a sub-controller. A network node may operate as one or both of a meta-controller and a sub-controller.

[0044] In some embodiments, the non-limiting terms wireless device (WD) or a user equipment (UE) are used interchangeably. The WD herein can be any type of wireless device capable of communicating with a network node or another WD over radio signals, such as wireless device (WD). The WD may also be a radio communication device, target device, device to device (D2D) WD, machine type WD or WD capable of machine to machine communication (M2M), low-cost and/or low-complexity WD, a sensor equipped with WD, Tablet, mobile terminals, smart phone, laptop embedded equipped (LEE), laptop mounted equipment (LME), USB dongles, Customer Premises Equipment (CPE), an Internet of Things (IoT) device, or a Narrowband IoT (NB-IoT) device, etc.

[0045] Also, in some embodiments the generic term radio network node is used. It can be any kind of a radio network node which may comprise any of base station, radio base station, base transceiver station, base station controller, network controller, RNC, evolved Node B (eNB), Node B, gNB, Multi-cell/multicast Coordination Entity (MCE), IAB node, relay node, access point, radio access point, Remote Radio Unit (RRU) Remote Radio Head (RRH).

[0046] Note that although terminology from one particular wireless system, such as, for example, 3GPP LTE and/or New Radio (NR), may be used in this disclosure, this should not be seen as limiting the scope of the disclosure to only the aforementioned system. Other wireless systems, including without limitation Wide Band Code Division Multiple Access (WCDMA), Worldwide Interoperability for Microwave Access (WiMax), Ultra Mobile Broadband (UMB) and Global System for Mobile Communications (GSM), may also benefit from exploiting the ideas covered within this disclosure.

[0047] Note further, that functions described herein as being performed by a wireless device or a network node may be distributed over a plurality of wireless devices and/or network nodes. In other words, it is contemplated that the functions of the network node and wireless device described herein are not limited to performance by a single physical device and, in fact, can be distributed among several physical devices.

[0048] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. It will be further understood that terms used herein should be interpreted as having a meaning that is consistent with their meaning in the context of this specification and the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0049] Some embodiments provide reconfigurable intelligent surface (RIS)-assisted energy-efficient (EE) radio access network (RAN) using hierarchical reinforcement learning.

[0050] Referring now to the drawing figures, in which like elements are referred to by like reference numerals, there is shown in FIG. 1 a schematic diagram of a communication system 10, according to an embodiment, such as a 3GPP-type cellular network that may support standards such as LTE and/or NR (5G), which comprises an access network 12, such as a radio access network, and a core network 14. The access network 12 comprises a plurality of network nodes 16a, 16b, 16c (referred to collectively as network nodes 16), such as NBs, eNBs, gNBs or other types of wireless access points, each defining a corresponding coverage area 18a, 18b, 18c (referred to collectively as coverage areas 18). Each network node 16a, 16b, 16c is connectable to the core network 14 over a wired or wireless connection 20. A first wireless device (WD) 22a located in coverage area 18a is configured to wirelessly connect to, or be paged by, the corresponding network node 16a. A second WD 22b in coverage area 18b is wirelessly connectable to the corresponding network node 16b. While a plurality of WDs 22a, 22b (collectively referred to as wireless devices 22) are illustrated in this example, the disclosed embodiments are equally applicable to a situation where a sole WD is in the coverage area or where a sole WD is connecting to the corresponding network node 16. Note that although only two WDs 22 and three network nodes 16 are shown for convenience, the communication system may include many more WDs 22 and network nodes 16. In some embodiments, the network node 16a may be configured to operate as a macro base station (MBS) and will be referred to herein as MBS 16a. In some embodiments, the network nodes 16b and 16c may be configured to operate as small base stations (SBSs) and will be referred to herein as SBS 16b and/or SBS 16c.

[0051] Also, it is contemplated that a WD 22 can be in simultaneous communication and/or configured to separately communicate with more than one network node 16 and more than one type of network node 16. For example, a WD 22 can have dual connectivity with a network node 16 that supports LTE and the same or a different network node 16 that supports NR. As an example, WD 22 can be in communication with an eNB for LTE/E-UTRAN and a gNB for NR/NG-RAN.

[0052] The communication system 10 may itself be connected to a host computer 24, which may be embodied in the hardware and/or software of a standalone server, a cloud-implemented server, a distributed server or as processing resources in a server farm. The host computer 24 may be under the ownership or control of a service provider, or may be operated by the service provider or on behalf of the service provider. The connections 26, 28 between the communication system 10 and the host computer 24 may extend directly from the core network 14 to the host computer 24 or may extend via an optional intermediate network 30. The intermediate network 30 may be one of, or a combination of more than one of, a public, private or hosted network. The intermediate network 30, if any, may be a backbone network or the Internet. In some embodiments, the intermediate network 30 may comprise two or more sub-networks (not shown).

[0053] The communication system of FIG. 1 as a whole enables connectivity between one of the connected WDs 22a, 22b and the host computer 24. The connectivity may be described as an over-the-top (OTT) connection. The host computer 24 and the connected WDs 22a, 22b are configured to communicate data and/or signaling via the OTT connection, using the access network 12, the core network 14, any intermediate network 30 and possible further infrastructure (not shown) as intermediaries. The OTT connection may be transparent in the sense that at least some of the participating communication devices through which the OTT connection passes are unaware of routing of uplink and downlink communications. For example, a network node 16 may not or need not be informed about the past routing of an incoming downlink communication with data originating from a host computer 24 to be forwarded (e.g., handed over) to a connected WD 22a. Similarly, the network node 16 need not be aware of the future routing of an outgoing uplink communication originating from the WD 22a towards the host computer 24.

[0054] A network node 16a configured to operate as an MBS may be configured to include a meta-controller 32 which is configured to generate a transmission control signal to configure a transmission status of the at least one sub-controller, the transmission control signal being based at least in part on machine learning reinforced by feedback from at least one WD. A network node 16b configured to operate as an SBS may be configured to include a sub-controller 34 which is configured to determine a transmission status of the sub-controller based at least in part on machine learning reinforced by feedback from at least one WD. In some embodiments, a network node may be configured with both the meta-controller 32 and the sub-controller 34.

[0055] Example implementations, in accordance with an embodiment, of the WD 22, network node 16 and host computer 24 discussed in the preceding paragraphs will now be described with reference to FIG. 2. In a communication system 10, a host computer 24 comprises hardware (HW) 38 including a communication interface 40 configured to set up and maintain a wired or wireless connection with an interface of a different communication device of the communication system 10. The host computer 24 further comprises processing circuitry 42, which may have storage and/or processing capabilities. The processing circuitry 42 may include a processor 44 and memory 46. In particular, in addition to or instead of a processor, such as a central processing unit, and memory, the processing circuitry 42 may comprise integrated circuitry for processing and/or control. e.g., one or more processors and/or processor cores and/or FPGAs (Field Programmable Gate Array) and/or ASICs (Application Specific Integrated Circuitry) adapted to execute instructions. The processor 44 may be configured to access (e.g., write to and/or read from) memory 46, which may comprise any kind of volatile and/or nonvolatile memory, e.g., cache and/or buffer memory and/or RAM (Random Access Memory) and/or ROM (Read-Only Memory) and/or optical memory and/or EPROM (Erasable Programmable Read-Only Memory).

[0056] Processing circuitry 42 may be configured to control any of the methods and/or processes described herein and/or to cause such methods, and/or processes to be performed, e.g., by host computer 24. Processor 44 corresponds to one or more processors 44 for performing host computer 24 functions described herein. The host computer 24 includes memory 46 that is configured to store data, programmatic software code and/or other information described herein. In some embodiments, the software 48 and/or the host application 50 may include instructions that, when executed by the processor 44 and/or processing circuitry 42, causes the processor 44 and/or processing circuitry 42 to perform the processes described herein with respect to host computer 24. The instructions may be software associated with the host computer 24.

[0057] The software 48 may be executable by the processing circuitry 42. The software 48 includes a host application 50. The host application 50 may be operable to provide a service to a remote user, such as a WD 22 connecting via an OTT connection 52 terminating at the WD 22 and the host computer 24. In providing the service to the remote user, the host application 50 may provide user data which is transmitted using the OTT connection 52. The user data may be data and information described herein as implementing the described functionality. In one embodiment, the host computer 24 may be configured for providing control and functionality to a service provider and may be operated by the service provider or on behalf of the service provider. The processing circuitry 42 of the host computer 24 may enable the host computer 24 to observe, monitor, control, transmit to and/or receive from the network node 16 and or the wireless device 22.

[0058] The communication system 10 further includes a network node 16 provided in a communication system 10 and including hardware 58 enabling it to communicate with the host computer 24 and with the WD 22. The hardware 58 may include a communication interface 60 for setting up and maintaining a wired or wireless connection with an interface of a different communication device of the communication system 10, as well as a radio interface 62 for setting up and maintaining at least a wireless connection 64 with a WD 22 located in a coverage area 18 served by the network node 16. The radio interface 62 may be formed as or may include, for example, one or more RF transmitters, one or more RF receivers, and/or one or more RF transceivers. The communication interface 60 may be configured to facilitate a connection 66 to the host computer 24. The connection 66 may be direct or it may pass through a core network 14 of the communication system 10 and/or through one or more intermediate networks 30 outside the communication system 10.

[0059] In the embodiment shown, the hardware 58 of the network node 16 further includes processing circuitry 68. The processing circuitry 68 may include a processor 70 and a memory 72. In particular, in addition to or instead of a processor, such as a central processing unit, and memory, the processing circuitry 68 may comprise integrated circuitry for processing and/or control, e.g., one or more processors and/or processor cores and/or FPGAs (Field Programmable Gate Array) and/or ASICs (Application Specific Integrated Circuitry) adapted to execute instructions. The processor 70 may be configured to access (e.g., write to and/or read from) the memory 72, which may comprise any kind of volatile and/or nonvolatile memory, e.g., cache and/or buffer memory and/or RAM (Random Access Memory) and/or ROM (Read-Only Memory) and/or optical memory and/or EPROM (Erasable Programmable Read-Only Memory).

[0060] Thus, the network node 16 further has software 74 stored internally in, for example, memory 72, or stored in external memory (e.g., database, storage array, network storage device, etc.) accessible by the network node 16 via an external connection. The software 74 may be executable by the processing circuitry 68. The processing circuitry 68 may be configured to control any of the methods and/or processes described herein and/or to cause such methods, and/or processes to be performed, e.g., by network node 16. Processor 70 corresponds to one or more processors 70 for performing network node 16 functions described herein. The memory 72 is configured to store data, programmatic software code and/or other information described herein. In some embodiments, the software 74 may include instructions that, when executed by the processor 70 and/or processing circuitry 68, causes the processor 70 and/or processing circuitry 68 to perform the processes described herein with respect to network node 16. For example, processing circuitry 68 of the network node 16 may include a meta-controller 32 which is configured to generate a transmission control signal to configure a transmission status of the at least one sub-controller, the transmission control signal being based at least in part on machine learning reinforced by feedback from at least one WD. The processing circuitry 68 of the network node 16 may include, in addition to or instead of the meta-controller 32, a sub-controller 34 which is configured to determine a transmission status of the sub-controller based at least in part on machine learning reinforced by feedback from at least one WD.

[0061] In some embodiments, the network node 16 may be in communication with a WD 22 directly and/or via a reconfigurable intelligent surface (RIS) 36. In some embodiments, the RIS 36 and at least one sub-controller 34 are collocated.

[0062] The communication system 10 further includes the WD 22 already referred to. The WD 22 may have hardware 80 that may include a radio interface 82 configured to set up and maintain a wireless connection 64 with a network node 16 serving a coverage area 18 in which the WD 22 is currently located. The radio interface 82 may be formed as or may include, for example, one or more RF transmitters, one or more RF receivers, and/or one or more RF transceivers.

[0063] The hardware 80 of the WD 22 further includes processing circuitry 84. The processing circuitry 84 may include a processor 86 and memory 88. In particular, in addition to or instead of a processor, such as a central processing unit, and memory, the processing circuitry 84 may comprise integrated circuitry for processing and/or control, e.g., one or more processors and/or processor cores and/or FPGAs (Field Programmable Gate Array) and/or ASICs (Application Specific Integrated Circuitry) adapted to execute instructions. The processor 86 may be configured to access (e.g., write to and/or read from) memory 88, which may comprise any kind of volatile and/or nonvolatile memory, e.g., cache and/or buffer memory and/or RAM (Random Access Memory) and/or ROM (Read-Only Memory) and/or optical memory and/or EPROM (Erasable Programmable Read-Only Memory).

[0064] Thus, the WD 22 may further comprise software 90, which is stored in, for example, memory 88 at the WD 22, or stored in external memory (e.g., database, storage array, network storage device, etc.) accessible by the WD 22. The software 90 may be executable by the processing circuitry 84. The software 90 may include a client application 92. The client application 92 may be operable to provide a service to a human or non-human user via the WD 22, with the support of the host computer 24. In the host computer 24, an executing host application 50 may communicate with the executing client application 92 via the OTT connection 52 terminating at the WD 22 and the host computer 24. In providing the service to the user, the client application 92 may receive request data from the host application 50 and provide user data in response to the request data. The OTT connection 52 may transfer both the request data and the user data. The client application 92 may interact with the user to generate the user data that it provides.

[0065] The processing circuitry 84 may be configured to control any of the methods and/or processes described herein and/or to cause such methods, and/or processes to be performed, e.g., by WD 22. The processor 86 corresponds to one or more processors 86 for performing WD 22 functions described herein. The WD 22 includes memory 88 that is configured to store data, programmatic software code and/or other information described herein. In some embodiments, the software 90 and/or the client application 92 may include instructions that, when executed by the processor 86 and/or processing circuitry 84, causes the processor 86 and/or processing circuitry 84 to perform the processes described herein with respect to WD 22.

[0066] In some embodiments, the inner workings of the network node 16, WD 22, and host computer 24 may be as shown in FIG. 2 and independently, the surrounding network topology may be that of FIG. 1.

[0067] In FIG. 2, the OTT connection 52 has been drawn abstractly to illustrate the communication between the host computer 24 and the wireless device 22 via the network node 16, without explicit reference to any intermediary devices and the precise routing of messages via these devices. Network infrastructure may determine the routing, which it may be configured to hide from the WD 22 or from the service provider operating the host computer 24, or both. While the OTT connection 52 is active, the network infrastructure may further take decisions by which it dynamically changes the routing (e.g., on the basis of load balancing consideration or reconfiguration of the network).

[0068] The wireless connection 64 between the WD 22 and the network node 16 is in accordance with the teachings of the embodiments described throughout this disclosure. One or more of the various embodiments improve the performance of OTT services provided to the WD 22 using the OTT connection 52, in which the wireless connection 64 may form the last segment. More precisely, the teachings of some of these embodiments may improve the data rate, latency, and/or power consumption and thereby provide benefits such as reduced user waiting time, relaxed restriction on file size, better responsiveness, extended battery lifetime, etc.

[0069] In some embodiments, a measurement procedure may be provided for the purpose of monitoring data rate, latency and other factors on which the one or more embodiments improve. There may further be an optional network functionality for reconfiguring the OTT connection 52 between the host computer 24 and WD 22, in response to variations in the measurement results. The measurement procedure and/or the network functionality for reconfiguring the OTT connection 52 may be implemented in the software 48 of the host computer 24 or in the software 90 of the WD 22, or both. In embodiments, sensors (not shown) may be deployed in or in association with communication devices through which the OTT connection 52 passes; the sensors may participate in the measurement procedure by supplying values of the monitored quantities exemplified above, or supplying values of other physical quantities from which software 48, 90 may compute or estimate the monitored quantities. The reconfiguring of the OTT connection 52 may include message format, retransmission settings, preferred routing etc.; the reconfiguring need not affect the network node 16, and it may be unknown or imperceptible to the network node 16. Some such procedures and functionalities may be known and practiced in the art. In certain embodiments, measurements may involve proprietary WD signaling facilitating the host computer's 24 measurements of throughput, propagation times, latency and the like. In some embodiments, the measurements may be implemented in that the software 48, 90 causes messages to be transmitted, in particular empty or dummy messages, using the OTT connection 52 while it monitors propagation times, errors, etc.

[0070] Thus, in some embodiments, the host computer 24 includes processing circuitry 42 configured to provide user data and a communication interface 40 that is configured to forward the user data to a cellular network for transmission to the WD 22. In some embodiments, the cellular network also includes the network node 16 with a radio interface 62. In some embodiments, the network node 16 is configured to, and/or the network node's 16 processing circuitry 68 is configured to perform the functions and/or methods described herein for preparing/initiating/maintaining/supporting/ending a transmission to the WD 22, and/or preparing/terminating/maintaining/supporting/ending in receipt of a transmission from the WD 22.

[0071] In some embodiments, the host computer 24 includes processing circuitry 42 and a communication interface 40 that is configured to a communication interface 40 configured to receive user data originating from a transmission from a WD 22 to a network node 16. In some embodiments, the WD 22 is configured to, and/or comprises a radio interface 82 and/or processing circuitry 84 configured to perform the functions and/or methods described herein for preparing/initiating/maintaining/supporting/ending a transmission to the network node 16, and/or preparing/terminating/maintaining/supporting/ending in receipt of a transmission from the network node 16.

[0072] Although FIGS. 1 and 2 show various units such as meta-controller 32, and sub-controller 34 as being within a respective processor or the same processor, it is contemplated that these units may be implemented such that a portion of the unit is stored in a corresponding memory within the processing circuitry. In other words, the units may be implemented in hardware or in a combination of hardware and software within the processing circuitry.

[0073] FIG. 3 is a flowchart illustrating an example method implemented in a communication system, such as, for example, the communication system of FIGS. 1 and 2, in accordance with one embodiment. The communication system may include a host computer 24, a network node 16 and a WD 22, which may be those described with reference to FIG. 2. In a first step of the method, the host computer 24 provides user data (Block S100). In an optional substep of the first step, the host computer 24 provides the user data by executing a host application, such as, for example, the host application 50 (Block S102). In a second step, the host computer 24 initiates a transmission carrying the user data to the WD 22 (Block S104). In an optional third step, the network node 16 transmits to the WD 22 the user data which was carried in the transmission that the host computer 24 initiated, in accordance with the teachings of the embodiments described throughout this disclosure (Block S106). In an optional fourth step, the WD 22 executes a client application, such as, for example, the client application 92, associated with the host application 50 executed by the host computer 24 (Block S108).

[0074] FIG. 4 is a flowchart illustrating an example method implemented in a communication system, such as, for example, the communication system of FIG. 1, in accordance with one embodiment. The communication system may include a host computer 24, a network node 16 and a WD 22, which may be those described with reference to FIGS. 1 and 2. In a first step of the method, the host computer 24 provides user data (Block S110). In an optional substep (not shown) the host computer 24 provides the user data by executing a host application, such as, for example, the host application 50. In a second step, the host computer 24 initiates a transmission carrying the user data to the WD 22 (Block S112). The transmission may pass via the network node 16, in accordance with the teachings of the embodiments described throughout this disclosure. In an optional third step, the WD 22 receives the user data carried in the transmission (Block S114). FIG. 5 is a flowchart illustrating an example method implemented in a communication system, such as, for example, the communication system of FIG. 1, in accordance with one embodiment. The communication system may include a host computer 24, a network node 16 and a WD 22, which may be those described with reference to FIGS. 1 and 2. In an optional first step of the method, the WD 22 receives input data provided by the host computer 24 (Block S116). In an optional substep of the first step, the WD 22 executes the client application 92, which provides the user data in reaction to the received input data provided by the host computer 24 (Block S118). Additionally or alternatively, in an optional second step, the WD 22 provides user data (Block S120). In an optional substep of the second step, the WD provides the user data by executing a client application, such as, for example, client application 92 (Block S122). In providing the user data, the executed client application 92 may further consider user input received from the user. Regardless of the specific manner in which the user data was provided, the WD 22 may initiate, in an optional third substep, transmission of the user data to the host computer 24 (Block S124). In a fourth step of the method, the host computer 24 receives the user data transmitted from the WD 22, in accordance with the teachings of the embodiments described throughout this disclosure (Block S126).

[0075] FIG. 6 is a flowchart illustrating an example method implemented in a communication system, such as, for example, the communication system of FIG. 1, in accordance with one embodiment. The communication system may include a host computer 24, a network node 16 and a WD 22, which may be those described with reference to FIGS. 1 and 2. In an optional first step of the method, in accordance with the teachings of the embodiments described throughout this disclosure, the network node 16 receives user data from the WD 22 (Block S128). In an optional second step, the network node 16 initiates transmission of the received user data to the host computer 24 (Block S130). In a third step, the host computer 24 receives the user data carried in the transmission initiated by the network node 16 (Block S132).

[0076] FIG. 7 is a flowchart of an example process in a network node 16 for reconfigurable intelligent surface (RIS)-assisted energy-efficient (EE) radio access network (RAN) using hierarchical reinforcement learning. One or more blocks described herein may be performed by one or more elements of network node 16 such as by one or more of processing circuitry 68 (including the meta-controller 32), processor 70, radio interface 62 and/or communication interface 60. Network node 16 such as via processing circuitry 68 and/or processor 70 and/or radio interface 62 and/or communication interface 60 is configured to generate a transmission control signal to configure a transmission status of the at least one sub-controller 34, the transmission control signal being based at least in part on machine learning reinforced by feedback from at least one WD 22 (Block S134). The process also includes transmitting the transmission control signal to the at least one sub-controller 34 (Block S136).

[0077] In some embodiments, the machine learning includes: generating at least one goal based at least in part on the feedback and transforming the at least one goal into at least one policy by which the at least one sub-controller 34 determines a transmission status. In some embodiments, the at least one goal includes increasing a gain of the link between the network node 16 and the WD 22 via a reconfigurable intelligent surface, RIS 36. In some embodiments, the gain is determined based at least in part on phase shifts applied to at least one reconfigurable intelligent surface, RIS 36. In some embodiments, the at least one goal includes decreasing power consumption of a network that includes the network node and the WD. In some embodiments, the at least one goal includes determining whether to change an on/off state of the at least one sub-controller.

[0078] FIG. 8 is a flowchart of an example process in a network node 16 for reconfigurable intelligent surface (RIS)-assisted energy-efficient (EE) radio access network (RAN) using hierarchical reinforcement learning. One or more blocks described herein may be performed by one or more elements of network node 16 such as by one or more of processing circuitry 68 (including the sub-controller 34), processor 70, radio interface 62 and/or communication interface 60. Network node 16 such as via processing circuitry 68 and/or processor 70 and/or radio interface 62 and/or communication interface 60 is configured to receive a transmission control signal (Block S138). The process also includes determining a transmission status of the sub-controller 34 based at least in part on machine learning reinforced by feedback from at least one WD 22 (Block S140).

[0079] In some embodiments, the method also includes receiving a policy from the meta-controller 32, and determining an intrinsic reward based at least in part on the policy. In some embodiments, the policy is based at least in part on an exploration of at least one goal to maximize a long term reward. In some embodiments, a state of the sub-controller 34 is defined by a Markov decision process.

[0080] FIG. 9 is a flowchart of an example process in a network node 16 for reconfigurable intelligent surface (RIS)-assisted energy-efficient (EE) radio access network (RAN) using hierarchical reinforcement learning. One or more blocks described herein may be performed by one or more elements of network node 16 such as by one or more of processing circuitry 68 (including the meta-controller 32), processor 70, radio interface 62 and/or communication interface 60. Network node 16 such as via processing circuitry 68 and/or processor 70 and/or radio interface 62 and/or communication interface 60 is configured to determine a state of the meta-controller 32, the state of the meta-controller 32 including traffic load ratios of the plurality of sub-controllers 34 (Block S142). The method also includes receiving an extrinsic reward, the extrinsic reward being based at least in part on an energy efficiency of a cell including the plurality of sub-controllers 34 (Block S144). The method also includes selecting a goal according to a policy, the selected goal being an on/off state of each of the plurality of sub-controllers 34, the policy being selected to increase the extrinsic reward (Block S146). The method further includes configuring the plurality of sub-controllers 34 with the selected goal and an indication of the policy for selecting the goal (Block S148).

[0081] According to this aspect, in some embodiments, the extrinsic reward is based at least in part a ratio of a sum of throughputs of network nodes in the cell to a sum of power consumptions of the network nodes 16 in the cell. In some embodiments, the extrinsic reward is further based at least in part on a penalty factor to avoid overloading and based at least in part on a number of network nodes 16 in the cell that are overloaded. In some embodiments, maximizing the extrinsic reward includes maximizing a throughput of a link between the network node and 16 a plurality of WDs 22 via a reconfigurable intelligent surface, RIS 36. In some embodiments, the policy is one of a greedy policy and an E-greedy policy. In some embodiments, the greedy policy provides a selected goal that increases the extrinsic reward when a random number exceeds a threshold and provides a randomly selected goal when the random number does not exceed the threshold.

[0082] FIG. 10 is a flowchart of an example process in a network node 16 for reconfigurable intelligent surface (RIS)-assisted energy-efficient (EE) radio access network (RAN) using hierarchical reinforcement learning. One or more blocks described herein may be performed by one or more elements of network node 16 such as by one or more of processing circuitry 68 (including the sub-controller 34), processor 70, radio interface 62 and/or communication interface 60. Network node 16 such as via processing circuitry 68 and/or processor 70 and/or radio interface 62 and/or communication interface 60 is configured to receive a goal and an indication of a policy from the meta-controller 32 (Block S150). The process includes receiving an intrinsic reward, the intrinsic reward being based at least in part on a ratio of a throughput of the sub-controller 34 to a transmission power of the sub-controller 34 (Block S152). The process also includes selecting an action based at least in part on the goal and according to the policy, the action including adjusting the transmission power of the sub-controller 34 to increase the intrinsic reward (Block S154).

[0083] According to this aspect, in some embodiments, the intrinsic reward is further based at least in part on a penalty factor to avoid overloading and based at least in part on a number of network nodes 16 in a cell that are overloaded. In some embodiments, maximizing the intrinsic reward includes maximizing a throughput of a link between the network node 16 and a plurality of WDs 22 via a reconfigurable intelligent surface, RIS 36. In some embodiments, the policy is one of a greedy policy and an e-greedy policy. In some embodiments, the greedy policy provides a selected action that increases the intrinsic reward when a random number exceeds a threshold and provides a randomly selected action when the random number does not exceed the threshold. In some embodiments, the goal is an on/off state of the sub-controller 34.

[0084] Having described the general process flow of arrangements of the disclosure and having provided examples of hardware and software arrangements for implementing the processes and functions of the disclosure, the sections below provide details and examples of arrangements for reconfigurable intelligent surface (RIS)-assisted energy-efficient (EE) radio access networks (RAN) using hierarchical reinforcement learning.

[0085] Some embodiments provide reconfigurable intelligent surface (RIS)-assisted energy-efficient (EE) radio access networks (RAN) using hierarchical reinforcement learning.

[0086] FIG. 11 shows a heterogeneous network environment 94 that includes one network node 16a operating as a macro base station (MBS) and having a meta-controller 32 and several network nodes 16b, 16c operating as small base stations (SBSs) and having sub-controllers 34. The SBSs 16b, 16c may switch to the sleep mode when the traffic demand drops, which will reduce the energy consumption. It is assumed the MBS 16a can take over the active wireless devices (WDs) 22 that were previously associated with those small cells. On the other hand, reconfigurable intelligent surfaces (RIS) 36 may be deployed to reflect the signal from the MBS 16a and mitigate the high penetration loss of direct transmissions. Energy may be saved in at least two ways: (i) the meta-controller 32 decides the on/off status of SBSs 16b, 16c when traffic load changes, and (ii) the sub-controller 34 decides the transmission power of active SBSs 16. Note that although reference is made to two SBSs 16b and 16c, there may be more than two SBSs configured to operate as described herein with respect to SBSs 16b, 16c.

[0087] It is assumed that WDs 22 can receive the signal from network nodes (NN) 16 by direct and indirect transmissions. The direct link is considered non-line-of-sight (NLOS) transmission due to the dense buildings in the urban area. The baseband equivalent channel between a network node 16 and a WD 22 is given by:

[00001]$\begin{array}{cc}{H}_{B,k}=g_{b,k}{h}_{b,k}& \left(A\right)\end{array}$

where g.sub.b,k is the path loss from NN 16 to WD 22, and h.sub.p,k is a complex Gaussian distributed random vector with zero mean and unit variance.

[0088] The indirect link consists of the NN-RIS and RIS-WD links. Given that RIS 36 are designed to be deployed on the top or surface of tall buildings, the NN-RIS link is assumed to be line-of-sight (LOS) transmission:

[00002]$\begin{array}{cc}{H}_{B,R}=g_{b,r}\left[h_1,h_2,.Math.,h_N\right]& \left(B\right)\end{array}$

where N is the number of RIS elements, g.sub.b,r is the path loss between BS and RIS, and h.sub.N is the phase difference.

[0089] Then, the RIS 36 is configured to reflect the signal to WDs 22 via a phase shift vector =[.sub.1, .sub.2, . . . , .sub.N], and each .sub.N=exp (j.sub.N), where exp is the natural exponential function, j is the square root of 1, and .sub.N is the phase shift of RIS element N.

[0090] Considering the complex environment surrounding the WD 22, the RIS-WD link is presumed to be NLOS transmission and:

[00003]$\begin{array}{cc}{H}_{R,k}=g_{b,r}\left[h_{1,k}^{},h_{2,k}^{},.Math.,h_{N,k}^{}\right]& \left(C\right)\end{array}$

where g.sub.b,r is the path loss between RIS and WD k, and

[00004]$h_{N,k}^{}$

is the small scale fading of the signal reflected by the RIS element N.

[0091] Finally, the channel gain from NN j to WD k is:

[00005]$\begin{array}{cc}{G}_{j,k}=.Math.H_{B,k}+H_{R,k}{H}_{B,R}^{\mathrm{Tr}}.Math.& \left(D\right)\end{array}$

where

[00006]$H_{B,R}^{\mathrm{Tr}}$

means the transpose matrix of H.sub.B,R.

[0092] For phase shift control, assume the channel state information (CSI) of a NN 16 and WDs 22 is perfectly shared between NN 16 and RIS 36 by wired transmissions such as optical fibers, and the CSI includes average channel gain, received signal phase and fading distribution. The RIS phase shift will take the same phase as H.sub.B,k, and the total received signal will be strengthened.

[0093] The energy consumption model for an SBS 16b, 16c is:

[00007]$\begin{array}{cc}{P}_{\mathrm{in}}\{\begin{array}{cc}{P}_0+P_{\mathrm{out}},& \mathrm{if}0<P_{\mathrm{out}}{P}_{\max }\\ P_{\mathrm{sleep}},& {\mathrm{if}P}_{\mathrm{out}}=0\end{array}& \left(E\right)\end{array}$

where P.sub.0 is the fixed power consumption, is the slope of load-dependent power consumption, P.sub.out is the transmission power, P.sub.max is the maximum transmission power, and P.sub.sleep is the constant power consumption in sleep mode.

[0094] An HRL algorithm may be applied to optimize the energy efficiency of the system. The HRL agent 96 consists of two controllers, namely a meta-controller 32 and a sub-controller 34. A Markov decision process (MDP) tuple<States, Action, Transition probability, Reward, Goal> is used to describe these two controllers. Based on the current state, the meta-controller 32 generates high-level goals for sub-controllers 34. Then, these goals are transformed into high-level policies by the critic 98. The critic 98 may be collocated with the meta-controller 32 in the MBS 16a and the functions of the critic 98 may be implemented by the processing circuitry 68 of the network node 16. Consequently, the sub-controller 34 chooses low-level actions according to high-level policies, and receives an intrinsic reward. The meta-controller 32 may receive an extrinsic reward from the environment, and select new goals for the sub-controller 34. An idea behind the HRL is to introduce hierarchical architecture in RL. In particular, the meta-controller 32 will produce high-level policies to guide the low-level action selection of the sub-controller 34.

[0095] The meta-controller 32 is responsible, among other things, for the high-level policies for the HRL agent 96. The MBS 16a includes a meta-controller 32, and its Markov decision processes (MDPs): [0096] State: The state of the meta-controller 32 consists of the traffic load ratio of SBSs 16b, 16c:

[00008]$\begin{array}{cc}{s}_{\mathrm{meta}}=\left\{d_{\mathrm{SBS},1},d_{\mathrm{SBS},2},.Math.,d_{\mathrm{SBS},j},.Math.,d_{\mathrm{SBS},N_{\mathrm{SBS}}}\right\}& \left(F\right)\end{array}$

Where d.sub.SBS,j is the traffic load ratio of SBS j (i.e., traffic demand (buffer size) of WD/bearer to maximum traffic load of the SBS j), which is assumed to be a tuned constant value to normalize the current traffic load of SBS.), and N.sub.SBS is the total number of SBSs 16b, 16c. [0097] Goal: With the traffic load status of the SBSs 16b, 16c, the MBS 16s can generate high-level policies for the SBSs 16b, 16c. The goal g.sub.meta is turning on/off the SBSs 16b, 16c:

[00009]$\begin{array}{cc}{g}_{\mathrm{meta}}=\left\{q_{\mathrm{SBS},1},q_{\mathrm{SBS},2},.Math.,q_{\mathrm{SBS},j},.Math.q_{\mathrm{SBS},N_{\mathrm{SBS}}}\right\}& \left(G\right)\end{array}$

where q.sub.SBS,j is a binary variable to indicate the on/off status of SBSs 16b, 16c. q.sub.SBS,j=1 means keeping SBS j active, otherwise q.sub.SBS,j=0 denotes turning off SBS j to save energy. [0098] Extrinsic reward: The meta-controller 32 focuses more on the overall performance of the whole cell. Accordingly, the extrinsic reward is given by the total energy efficiency of the whole cell:

[00010]$\begin{array}{cc}{r}_{\mathrm{ex}}=\frac{{.Math.}_{j=1}^{N_{\mathrm{BS}}}{W}_j}{{.Math.}_{j=1}^{N_{\mathrm{BS}}}{P}_j}-n_{\mathrm{od}}& \left(H\right)\end{array}$

where NBs is the total number of BSs (network nodes 16) in the cell, W.sub.j is the throughput of BS j, P.sub.j is the power consumption of BS j, n.sub.od is the number of BSs 16 that are overloaded. Here, is a penalty factor to prevent the overloading. Overload occurs, for example, when the channel capacity of the network node 16 is unable to satisfy the demand of WDs 22. [0099] For the sub-controller 34, the Q-values are updated by:

[00011]$\begin{array}{cc}{Q}_{\mathrm{meta}}^{\mathrm{new}}\left(s_{\mathrm{meta}},g_{\mathrm{meta}}\right)=\left(1-\right)Q_{\mathrm{meta}}^{\mathrm{old}}\left(s_{\mathrm{meta}},g_{\mathrm{meta}}\right)+\left(r_{\mathrm{ex}}+\underset{g_{\mathrm{meta}}}{\max }\left(Q_{\mathrm{meta}}^{\mathrm{old}}\left(s_{\mathrm{meta}}^{},g_{\mathrm{meta}}\right)\right)\right)& \left(I\right)\end{array}$

where s.sub.meta denotes the next state, a is the learning rate, and is the discount function (0<<1,0<<1).

[00012]$Q_{\mathrm{meta}}^{\mathrm{old}}\mathrm{and}{Q}_{\mathrm{meta}}^{\mathrm{new}}$

denote old and new Q-values for meta-controller 32, which means the accumulated reward. [0100] Then an -greedy policy for the goal selection may be used:

[00013]$\begin{array}{cc}g=\{\begin{array}{cc}\arg \underset{g}{\max }Q\left(s_{\mathrm{meta}},g\right),& \mathrm{if}\mathrm{rand}>\\ \mathrm{random}\mathrm{goal}\mathrm{selection},& \mathrm{if}\mathrm{rand}\end{array}& \left(J\right)\end{array}$

where rand is a random number between 0 and 1, and 0<<1 is a preset fixed number. E-greedy policy can balance the exploration and exploitation of goals to maximize the long-term reward. [0101] Then, each SBS 16b, 16c is regarded as a sub-controller 34. The MDP of the sub-controller 34 may be defined by: [0102] State: The state s.sub.sub of SBS j is defined by its traffic load ratio s.sub.sub={d.sub.SBS}, which is given by:

[00014]$\begin{array}{cc}{d}_{\mathrm{SBS}}=\frac{D_{j,t}}{D_{j,\max }}& \left(K\right)\end{array}$

where D.sub.j,t is the traffic load at time slot t. D.sub.j,max is the max traffic load of SBS j, which is considered a constant value to normalize the current traffic load of SBS j. Here it may be assumed that the daily traffic load follows fixed patterns, and the max traffic load can be extracted by observing the daily traffic volume. [0103] Action: Based on s.sub.sub, the SBS 16b, 16c may change its transmission power P.sub.SBS to adapt to the traffic demand. Then, the action is defined by s.sub.sub={P.sub.SBS}. [0104] Intrinsic reward: The intrinsic reward rin of sub-controller 34 is:

[00015]$\begin{array}{cc}{r}_{\mathrm{in}}=\frac{W_j}{P_{\mathrm{SBS}}}-& \left(L\right)\end{array}$

where W.sub.j is the throughput of SBS j, t is the penalty for overloading. Here the overloading means that the current traffic demand has exceeded the transmission capability of one SBS 16b, 16c, and then the attached WDs 22 may experience a long delay. With this definition, the sub-controller 34 aims at maximizing its own EE and preventing overloading. [0105] The Q-values of sub-controller 34 are updated by:

[00016]$\begin{array}{cc}{Q}_{\mathrm{sub}}^{\mathrm{new}}\left(s_{\mathrm{meta}},g_{\mathrm{meta}},a_{\mathrm{sub}}\right)=\left(1-\right)Q_{\mathrm{sub}}^{\mathrm{old}}\left(s_{\mathrm{sub}},g_{\mathrm{meta}},a_{\mathrm{sub}}\right)+\left(r_{\mathrm{in}}+\underset{a_{\mathrm{sub}}}{\max }\left(Q_{\mathrm{sub}}^{\mathrm{old}}\left(s_{\mathrm{sub}}^{},g_{\mathrm{meta}}^{},a_{\mathrm{sub}}\right)\right)\right)& \left(M\right)\end{array}$

where

[00017]$s_{\mathrm{sub}}^{}$

is the next state,

[00018]$g_{\mathrm{meta}}^{}$

is the next goal generated by meta-controller 32.

[00019]$Q_{\mathrm{sub}}^{\mathrm{old}}\mathrm{and}{Q}_{\mathrm{sub}}^{\mathrm{new}}$

denote old and new Q-values for sub-controller 34. [0106] Similarly, -greedy policy may be used for the goal selection:

[00020]$\begin{array}{cc}a=\{\begin{array}{cc}\arg \underset{a}{\max }Q\left(s_{\mathrm{sub}},g_{\mathrm{meta}},a\right),& \mathrm{if}\mathrm{rand}>\\ \mathrm{random}\mathrm{action}\mathrm{selection},& \mathrm{if}\mathrm{rand}\end{array}& \left(N\right)\end{array}$ [0107] Note that the discount factor , learning rate , and value in equation (J) and (N) are all tunable parameters in the proposed solution. Here, a grid-search method may be deployed by trying different configurations, and selecting the best parameter combinations accordingly. [0108] A flowchart of an example process according to principles set forth herein is shown in FIG. 12. During an exploration phase (Block S156), the meta-controller 32 selects a goal using an e-greedy policy (Block S158). The sub-controllers 34 select actions using the -greedy policy (Block S160). When exploration is completed (Block S156), the meta-controller 32 selects a goal using a greedy policy (Block S162), and the sub-controllers 34 select actions using the greedy policy (Block S164). After the goals and actions have been selected, they are implemented (Block S166), and states of the meta-controller 32 and the sub-controllers 34 are updated (Block S168). Also, the intrinsic and extrinsic rewards are computed (Block S170). The Q-values of the meta-controller 32 and the sub-controllers 34 may be updated. When the maximum number of iterations are performed (Block S174), the optimal goals and actions are output (Block S176). If the maximum number of iterations have not been performed (Block S174), then the process continues at Block S156. [0109] Some embodiments may include one or more of the following steps: [0110] Step 1: Initializing all system status. Setting configurable parameters including: exploration iteration numbers, maximum iteration number. [0111] Step 2: Based on predefined exploration time, decide the goal and action selection strategies, i.e., greedy policy or e-greedy policy using equations (J) and (N). [0112] Step 3: After goals and actions are selected and implemented, the system state is updated, and the intrinsic and extrinsic rewards are calculated using equations (H) and (L). [0113] Step 4: Then update the Q-values of the meta-controller 32 and sub-controller 34 as equations (I) and (M), respectively. [0114] Step 5: If the HRL agent 96 reaches maximum iterations, then it will produce the best goal and action selection sequence. Otherwise, it will repeat Steps 1 to 4 until the maximum iteration number. [0115] An example of information flow between nodes is shown in FIG. 13. The meta-controller 32 may send a sleep control signal to a plurality of sub-controllers 34. The channel state information (CSI) of network node-to-wireless connections may be employed to configure the RIS 36. The meta-controller 32 and the sub-controllers 34 may share network performance information and/or CSI feedback from the WDs 22 in the network environment. One or more of the WDs 22 may be configured to receive downlink packet transmission from one or more of the meta-controller 32, a sub-controller 34 and the RIS 36. [0116] Some embodiments may include one or more of the following: [0117] Embodiment A1. A network node operating as a meta-controller and configured to communicate with a wireless device (WD) and at least one sub-controller, the network node configured to, and/or comprising a radio interface and/or comprising processing circuitry configured to: [0118] generate a transmission control signal to configure a transmission status of the at least one sub-controller, the transmission control signal being based at least in part on machine learning reinforced by feedback from at least one WD; and [0119] transmit the transmission control signal to the at least one sub-controller. [0120] Embodiment A2. The network node of Embodiment A1, wherein the machine learning includes: [0121] generating at least one goal based at least in part on the feedback; [0122] transforming the at least one goal into at least one policy by which the at least one sub-controller determines a transmission status. [0123] Embodiment A3. The network node of Embodiment A2, wherein the at least one goal includes increasing a gain of the link between the network node and the WD via a reconfigurable intelligent surface, RIS. [0124] Embodiment A4. The network node of Embodiment A3, wherein the gain is determined based at least in part on phase shifts applied to at least one reconfigurable intelligent surface, RIS. [0125] Embodiment A5. The network node of Embodiment A2, wherein the at least one goal includes decreasing power consumption of a network that includes the network node and the WD. [0126] Embodiment A6. The network node of any of Embodiments A2-A5, wherein the at least one goal includes determining whether to change an on/off state of the at least one sub-controller. [0127] Embodiment B1. A method implemented in a network node operating as a meta-controller and configured to communicate with a wireless device and at least one sub-controller, WD, the method comprising: [0128] generating a transmission control signal to configure a transmission status of the at least one sub-controller, the transmission control signal being based at least in part on machine learning reinforced by feedback from at least one WD; and [0129] transmitting the transmission control signal to the at least one sub-controller. Embodiment B2. The method of Embodiment B1, wherein the machine learning includes: [0130] generating at least one goal based at least in part on the feedback; [0131] transforming the at least one goal into at least one policy by which the at least one sub-controller determines a transmission status. [0132] Embodiment B3. The method of Embodiment B2, wherein the at least one goal includes increasing a gain of the link between the network node and the WD via a reconfigurable intelligent surface, RIS. [0133] Embodiment B4. The method of Embodiment B3, wherein the gain is determined based at least in part on phase shifts applied to at least one reconfigurable intelligent surface, RIS. [0134] Embodiment B5. The method of Embodiment B2, wherein the at least one goal includes decreasing power consumption of a network that includes the network node and the WD. [0135] Embodiment B6. The method of any of Embodiments B2-B5, wherein the at least one goal includes determining whether to change an on/off state of the at least one sub-controller. [0136] Embodiment C1. A network node configured to operate as a sub-controller configured to communicate with a wireless device (WD) and at least one network node operating as a meta-controller, the network node configured to, and/or comprising a radio interface and/or comprising processing circuitry configured to: [0137] receive a transmission control signal; and [0138] determine a transmission status of the sub-controller based at least in part on machine learning reinforced by feedback from at least one WD. [0139] Embodiment C2. The network node of Embodiment C1, wherein the network node, radio interface and/or processing circuitry are further configured to: [0140] receive a policy from the meta-controller; and [0141] determine an intrinsic reward based at least in part on the policy. [0142] Embodiment C3. The network node of Embodiment C2, wherein the policy is based at least in part on an exploration of at least one goal to maximize a long term reward. [0143] Embodiment C4. The network node of any of Embodiments C1-C3, wherein a state of the sub-controller is defined by a Markov decision process. [0144] Embodiment D1. A method implemented in a network node operating as a sub-controller and configured to communicate with a wireless device (WD) and at least one network node operating as a meta-controller, the method comprising: [0145] receiving a transmission control signal; and [0146] determining a transmission status of the sub-controller based at least in part on machine learning reinforced by feedback from at least one WD. [0147] Embodiment D2. The method of Embodiment D1, further comprising: [0148] receiving a policy from the meta-controller; and [0149] determining an intrinsic reward based at least in part on the policy. [0150] Embodiment D3. The method of Embodiment D2, wherein the policy is based at least in part on an exploration of at least one goal to maximize a long term reward. [0151] Embodiment D4. The method of any of Embodiments D1-D3, wherein a state of the sub-controller is defined by a Markov decision process. [0152] As will be appreciated by one of skill in the art, the concepts described herein may be embodied as a method, data processing system, computer program product and/or computer storage media storing an executable computer program. Accordingly, the concepts described herein may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects all generally referred to herein as a circuit or module. Any process, step, action and/or functionality described herein may be performed by, and/or associated to, a corresponding module, which may be implemented in software and/or firmware and/or hardware. Furthermore, the disclosure may take the form of a computer program product on a tangible computer usable storage medium having computer program code embodied in the medium that can be executed by a computer. Any suitable tangible computer readable medium may be utilized including hard disks, CD-ROMs, electronic storage devices, optical storage devices, or magnetic storage devices. [0153] Some embodiments are described herein with reference to flowchart illustrations and/or block diagrams of methods, systems and computer program products. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer (to thereby create a special purpose computer), special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. [0154] These computer program instructions may also be stored in a computer readable memory or storage medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instruction means which implement the function/act specified in the flowchart and/or block diagram block or blocks. [0155] The computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. [0156] It is to be understood that the functions/acts noted in the blocks may occur out of the order noted in the operational illustrations. For example, two blocks shown in succession may in fact be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality/acts involved. Although some of the diagrams include arrows on communication paths to show a primary direction of communication, it is to be understood that communication may occur in the opposite direction to the depicted arrows. [0157] Computer program code for carrying out operations of the concepts described herein may be written in an object oriented programming language such as Python, Java or C++. However, the computer program code for carrying out operations of the disclosure may also be written in conventional procedural programming languages, such as the C programming language. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer. In the latter scenario, the remote computer may be connected to the user's computer through a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). [0158] Many different embodiments have been disclosed herein, in connection with the above description and the drawings. It will be understood that it would be unduly repetitious and obfuscating to literally describe and illustrate every combination and subcombination of these embodiments. Accordingly, all embodiments can be combined in any way and/or combination, and the present specification, including the drawings, shall be construed to constitute a complete written description of all combinations and subcombinations of the embodiments described herein, and of the manner and process of making and using them, and shall support claims to any such combination or subcombination. [0159] Abbreviations that may be used in the preceding description include:

TABLE-US-00001 Abbreviation Explanation BS Base stations CSI Channel state information EE Energy efficiency HRL Hierarchical reinforcement learning IOT Internet of things LOS Line-of-sight MBS Main base stations MDP Markov decision process NLOS non-line-of-sight MBS Main base stations RAN Radio access network MBS Main base stations RL Reinforcement learning SBS Small base stations SINR Signal-to-interference-plus-noise ratio UE User equipment [0160] It will be appreciated by persons skilled in the art that the embodiments described herein are not limited to what has been particularly shown and described herein above. In addition, unless mention was made above to the contrary, it should be noted that all of the accompanying drawings are not to scale. A variety of modifications and variations are possible in light of the above teachings without departing from the scope of the following claims.