python – pytorch LSTM model with unequal hidden layer

i have tuned a lstm model in keras as follows. but i dont know how write that code in pytorch. i put my pytorch code here but i dont think be right, because It does not give the right answer. how much I searched, I could not find a sample code in pytorch for more than one lstm layer with unequal hidden layers. my input shape is (None,(60,10)) with output shape (None,15) Please express a similar example for my keras model in pytorch. Thank

my_Keras_model:

model_input = keras.Input(shape=(60, 10))
x_1 = layers.LSTM(160,return_sequences=True)(model_input)
x_1 = layers.LSTM(190)(x_1)
x_1 = layers.Dense(200)(x_1)
x_1 = layers.Dense(15)(x_1)
model = keras.models.Model(model_input, x_1)

my_pytorch_model:

input_dim = 10
hidden_dim_1 = 160
hidden_dim_2 = 190
hidden_dim_3 = 200
num_layers = 1
output_dim = 15

class LSTM(nn.Module):
    def __init__(self, input_dim, hidden_dim_1, hidden_dim_2, hidden_dim_3 ,num_layers, output_dim):
        super(LSTM, self).__init__()
        self.hidden_dim_1 = hidden_dim_1
        self.hidden_dim_2 = hidden_dim_2
        self.hidden_dim_3 = hidden_dim_3
        self.num_layers = num_layers
        
        self.lstm_1 = nn.LSTM(input_dim, hidden_dim_1, num_layers, batch_first=True)
        self.lstm_2 = nn.LSTM(hidden_dim_1, hidden_dim_2, num_layers, batch_first=True)
        self.fc_1 = nn.Linear(hidden_dim_2, hidden_dim_3)
        self.fc_out = nn.Linear(hidden_dim_3, output_dim)

    def forward(self, x):
        input_X = x
        h_1 = torch.zeros(num_layers, 1 , self.hidden_dim_1).requires_grad_()
        c_1 = torch.zeros(num_layers, 1 , self.hidden_dim_1).requires_grad_()
        h_2 = torch.zeros(num_layers, 1 , self.hidden_dim_2).requires_grad_()
        c_2 = torch.zeros(num_layers, 1 , self.hidden_dim_2).requires_grad_()
        out_put = ()
        
        for i, input_t in enumerate(input_X.chunk(input_X.size(0))):
          out_lstm_1 , (h_1, c_1) = self.lstm_1(input_t, (h_1.detach(), c_1.detach()))
          out_lstm_2 , (h_2, c_2) = self.lstm_2(out_lstm_1, (h_2.detach(), c_2.detach()))
          out_Dense_1 = self.fc_1(out_lstm_2(:, -1, :))
          out_Dense_out = self.fc_out(out_Dense_1)
          out_put += out_Dense_out
        out_put = torch.stack(out_put, 0).squeeze(1)
        return out_put

python – Initialize weights in pytorch with customized weight?

I want to initialize the weights of my neural network with parameters samples from a specific distribution, that is not already present in nn.init module. Let’s say for example a beta distribution.

I saw that you can use the method described there: Custom weight initialization in PyTorch

But there you need to use the nn.init. What is an elegant way to initialize weights from a distribution that I specify

python – No puedo usar pytorch 11.1 con GPU, usando una NVIDIA 730 GT, que debo hacer

Use GPU-Z para obtener las especificaciones de mi GPU, y su controlador en este caso 491.92

introducir la descripción de la imagen aquí

Aparentemente tendria que estar todo bien, no?
introducir la descripción de la imagen aquí

Luego instale la GPU-accelerated library of primitives for DL, NVIDIA cuDNN en su version…
introducir la descripción de la imagen aquí

Esta version que es compatible (en teoria), para CUDA 11.0, 11.1 y 11.2
introducir la descripción de la imagen aquí

Se que se debe escoger el pytorch en funcion del CUDA que quieras instalar, pero en este caso se que usare el pytorch para la 11.1 osea que elegi esa version.

Tambien instale el CUDA Toolkit 11.1.0, que creo en mi caso es el que es consistente con el resto pero estoy en dudas. Aun asi aqui dejo el link de donde lo baje con el exe(local).

https://developer.nvidia.com/cuda-11.1.0-download-archive?target_os=Windows&target_arch=x86_64&target_version=10

introducir la descripción de la imagen aquí

Ahora instale el pytorch para la version 11.1 (que es la que queria) desde el gestor pip, simplemente poniendo el siguiente code copiado de la page:

pip install torch==1.8.1+cu111 torchvision==0.9.1+cu111 torchaudio===0.8.1 -f https://download.pytorch.org/whl/torch_stable.html

introducir la descripción de la imagen aquí

Estube probando pytorch en consola con la impresion de un tensor, y aparentemente funciona perfecto, pero claro hasta ahora con eso solo pruebo que funcione torch con la CPU, ya que no especifique el device.

>>> import torch
>>> x = torch.rand(5, 3)
>>> print(x)
tensor(((0.1242, 0.4253, 0.9530),
        (0.2290, 0.8633, 0.2871),
        (0.3668, 0.5047, 0.7253),
        (0.9148, 0.0506, 0.3024),
        (0.3645, 0.1265, 0.1900)))

Luego ejecute esto:

import torch
print(torch.cuda.is_available())

Y me devolvio True, a lo que entiendo que CUDA si funciona (pero no es asi)

Estuve viendo gente a la que le ocurrio algo similar, pero no me funcionan las soluciones que plantean(o porque estan desactualizadas las soluciones, o no se hacerlo bien). Ellos dicen que instale pytorch desde el codigo fuente o algo asi…

Aun asi creo que el problema es pytorch.
y el cuda cc, imagino que debe ser un compiler pero no lo se con seguridad, que dicen?

En el siguiente link, plantean una “guia de instalacion algo complicada para mi al menos”

introducir la descripción de la imagen aquí

https://github.com/pytorch/pytorch#from-source

Fui a ese repositorio de github y descargue el proyecto a mi pc.

Intente ejecutar ese setup.py con torch anterior eliminado y sin torch anterior eliminado, y tira…

(base) C:UsersMIPCDesktopMATIVtuber_HPpytorch-master>python setup.py
Building wheel torch-1.9.0a0+gitUnknown
usage: setup.py (global_opts) cmd1 (cmd1_opts) (cmd2 (cmd2_opts) ...)
   or: setup.py --help (cmd1 cmd2 ...)
   or: setup.py --help-commands
   or: setup.py cmd --help

error: no commands supplied

Realmente no entiendo para que es eso…

Lo que me sigue dejando en duda es eso del compilador que pide en C++
Y respecto al CUDA Toolkit 11.1 y el NVIDIA cudDNN (en version 11.1) en teoria los podria dejar asi… como mostre que les instale mas arriba, no?

introducir la descripción de la imagen aquí

introducir la descripción de la imagen aquí

De todos modos, al no poder usar con GPU, adapte mi proyecto a CPU modificando todo lo que diga to_gpu o to_device, y andubo con CPU usando los 3 en 11.1, pero como CPU (lento, muy lento, per andubo, osea que con eso ya descarto que sea mi proyecto)

Si lo ejecuto con GPU, usando el supuesto CUDA 11.1 instalado me tira estos errores, y ahi el problema:

(base) C:UsersMIPCDesktopMATIVtuber_HPVtuberProjectAssetsTrackingBackend>python main.py
starting up on 127.0.0.1 port 65432
Using cache found in C:UsersMIPC/.cachetorchhubintel-isl_MiDaS_master
Loading weights:  None
Using cache found in C:UsersMIPC/.cachetorchhubfacebookresearch_WSL-Images_master
Using cache found in C:UsersMIPC/.cachetorchhubintel-isl_MiDaS_master
waiting for a connection
connection from ('127.0.0.1', 13676)
Connection closed
Traceback (most recent call last):
  File "main.py", line 39, in <module>
    pose_data = pose_estimator.get_pose_data(img.copy())
  File "C:UsersMIPCDesktopMATIVtuber_HPVtuberProjectAssetsTrackingBackendutilspose_estimator.py", line 74, in get_pose_data
    heatmaps, pafs, scale, pad = self.infer_fast(img)
  File "C:UsersMIPCDesktopMATIVtuber_HPVtuberProjectAssetsTrackingBackendutilspose_estimator.py", line 49, in infer_fast
    stages_output = self.net(tensor_img)
  File "C:UsersMIPCanaconda3libsite-packagestorchnnmodulesmodule.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:UsersMIPCDesktopMATIVtuber_HPVtuberProjectAssetsTrackingBackendemotion_modelswith_mobilenet.py", line 134, in forward
    backbone_features = self.model(x)
  File "C:UsersMIPCanaconda3libsite-packagestorchnnmodulesmodule.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
File "C:UsersMIPCanaconda3libsite-packagestorchnnmodulescontainer.py", line 119, in forward
    input = module(input)
  File "C:UsersMIPCanaconda3libsite-packagestorchnnmodulesmodule.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:UsersMIPCanaconda3libsite-packagestorchnnmodulescontainer.py", line 119, in forward
    input = module(input)
  File "C:UsersMIPCanaconda3libsite-packagestorchnnmodulesmodule.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:UsersMIPCanaconda3libsite-packagestorchnnmodulesconv.py", line 399, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "C:UsersMIPCanaconda3libsite-packagestorchnnmodulesconv.py", line 395, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: CUDA error: no kernel image is available for execution on the device
( WARN:1) global C:UsersappveyorAppDataLocalTemp1pip-req-build-kh7iq4w7opencvmodulesvideoiosrccap_msmf.cpp (434) `anonymous-namespace'::SourceReaderCB::~SourceReaderCB terminating async callback

Trate de describir d ela mejor manera que pude todo lo que hice haber si ustedes encuentran el error ­čÖü , pero sigue sin funcionar…
Probe si la camara es correcta y opencv la detecta y da video streaming osea que un problema con la webcam esta descartado.

Aun asi sigue tirando esto…

RuntimeError: CUDA error: no kernel image is available for execution on the device
( WARN:1) global C:UsersappveyorAppDataLocalTemp1pip-req-build-kh7iq4w7opencvmodulesvideoiosrccap_msmf.cpp (434) `anonymous-namespace'::SourceReaderCB::~SourceReaderCB terminating async callback

Ya no se mas que hacer para hacer funcionar a pytorch en mi pc, espero realmente puedan ayudarme. Como veran trate de explicarme lo mejor posible, pero encerio que no se mas que hacerle.

Need new GLIBC on Centos 6 to use PyTorch on GPUs

I am using a supercomputer facility which is running Centos 6. The node I want to use has 3 Tesla V100. The problem is that the version of GLIBC installed on that node is not compatible with the latest versions of PyTorch I must use.

I do not have root. Hence, I need a way to use another GLIBC from a user level. I can talk to the sysadmins and do stuff from root (like use Docker, or something like that) but I cannot reinstall OS or GLIBC globally. I have tried to install GLIBC by myself without root, but could not do it right; it did not work. It takes too much time, and cannot find a tutorial to do it right.

I have some ideas in mind, like trying to run a Container that can access the node and use other OS; when my scheduled computing time ends I can leave the node the same way I got it; for the next user on the supercomputer facility.

I was also thinking about chroot; Download an ISO of Centos 8 and chroot into it, but I do not know if it would use the GLIBC used by the Host OS, or Centos 8 GLIBC.

What do you recommend me to do? Do you think Docker would suit my needs? Or other containerization solution? Keep trying with installing other GLIBC?

python – validation and test loss for a variety of PyTorch time series forecasting models

Hi everyone I’m trying to reduce the complexity of some of my Python code. The function below aims to compute the validation and test loss for a variety of PyTorch time series forecasting models. I won’t go into all the intricacies but needs to support models that return multiple targets, an output distribution + std (as opposed to a single tensor), and models that require masked elements of the target sequence. This over time has resulted in long if else blocks and lots of other bad practices.

I’ve used dictionaries before to map long if else statements but due to the nested nature of this code it doesn’t seem like it would work well here. I also don’t really see the point in just creating more functions as that just moves the if else statements somewhere else and requires passing more parameters around. Does anyone have any ideas? There are several unit tests that run from the different paths in this code now. However, it is still cumbersome to read. Plus soon I will have even more model variations to expand and support. Full code can in context can be seen at this link.

def compute_validation(validation_loader: DataLoader,
                       model,
                       epoch: int,
                       sequence_size: int,
                       criterion: Type(torch.nn.modules.loss._Loss),
                       device: torch.device,
                       decoder_structure=False,
                       meta_data_model=None,
                       use_wandb: bool = False,
                       meta_model=None,
                       multi_targets=1,
                       val_or_test="validation_loss",
                       probabilistic=False) -> float:
    """Function to compute the validation loss metrics

    :param validation_loader: The data-loader of either validation or test-data
    :type validation_loader: DataLoader
    :param model: model
    :type model: (type)
    :param epoch: The epoch where the validation/test loss is being computed.
    :type epoch: int
    :param sequence_size: The number of historical time steps passed into the model
    :type sequence_size: int
    :param criterion: The evaluation metric function
    :type criterion: Type(torch.nn.modules.loss._Loss)
    :param device: The device
    :type device: torch.device
    :param decoder_structure: Whether the model should use sequential decoding, defaults to False
    :type decoder_structure: bool, optional
    :param meta_data_model: The model to handle the meta-data, defaults to None
    :type meta_data_model: PyTorchForecast, optional
    :param use_wandb: Whether Weights and Biases is in use, defaults to False
    :type use_wandb: bool, optional
    :param meta_model: Whether the model leverages meta-data, defaults to None
    :type meta_model: bool, optional
    :param multi_targets: Whether the model, defaults to 1
    :type multi_targets: int, optional
    :param val_or_test: Whether validation or test loss is computed, defaults to "validation_loss"
    :type val_or_test: str, optional
    :param probabilistic: Whether the model is probablistic, defaults to False
    :type probabilistic: bool, optional
    :return: The loss of the first metric in the list.
    :rtype: float
    """
    print('Computing validation loss')
    unscaled_crit = dict.fromkeys(criterion, 0)
    scaled_crit = dict.fromkeys(criterion, 0)
    model.eval()
    output_std = None
    multi_targs1 = multi_targets
    scaler = None
    if validation_loader.dataset.no_scale:
        scaler = validation_loader.dataset
    with torch.no_grad():
        i = 0
        loss_unscaled_full = 0.0
        for src, targ in validation_loader:
            src = src if isinstance(src, list) else src.to(device)
            targ = targ if isinstance(targ, list) else targ.to(device)
            # targ = targ if isinstance(targ, list) else targ.to(device)
            i += 1
            if decoder_structure:
                if type(model).__name__ == "SimpleTransformer":
                    targ_clone = targ.detach().clone()
                    output = greedy_decode(
                        model,
                        src,
                        targ.shape(1),
                        targ_clone,
                        device=device)(
                        :,
                        :,
                        0)
                elif type(model).__name__ == "Informer":
                    multi_targets = multi_targs1
                    filled_targ = targ(1).clone()
                    pred_len = model.pred_len
                    filled_targ(:, -pred_len:, :) = torch.zeros_like(filled_targ(:, -pred_len:, :)).float().to(device)
                    output = model(src(0).to(device), src(1).to(device), filled_targ.to(device), targ(0).to(device))
                    labels = targ(1)(:, -pred_len:, 0:multi_targets)
                    src = src(0)
                    multi_targets = False
                else:
                    output = simple_decode(model=model,
                                           src=src,
                                           max_seq_len=targ.shape(1),
                                           real_target=targ,
                                           output_len=sequence_size,
                                           multi_targets=multi_targets,
                                           probabilistic=probabilistic,
                                           scaler=scaler)
                    if probabilistic:
                        output, output_std = output(0), output(1)
                        output, output_std = output(:, :, 0), output_std(0)
                        output_dist = torch.distributions.Normal(output, output_std)
            else:
                if probabilistic:
                    output_dist = model(src.float())
                    output = output_dist.mean.detach().numpy()
                    output_std = output_dist.stddev.detach().numpy()
                else:
                    output = model(src.float())
            if multi_targets == 1:
                labels = targ(:, :, 0)
            elif multi_targets > 1:
                labels = targ(:, :, 0:multi_targets)
            validation_dataset = validation_loader.dataset
            for crit in criterion:
                if validation_dataset.scale:
                    # Should this also do loss.item() stuff?
                    if len(src.shape) == 2:
                        src = src.unsqueeze(0)
                    src1 = src(:, :, 0:multi_targets)
                    loss_unscaled_full = compute_loss(labels, output, src1, crit, validation_dataset,
                                                      probabilistic, output_std, m=multi_targets)
                    unscaled_crit(crit) += loss_unscaled_full.item() * len(labels.float())
                loss = compute_loss(labels, output, src, crit, False, probabilistic, output_std, m=multi_targets)
                scaled_crit(crit) += loss.item() * len(labels.float())
    if use_wandb:
        if loss_unscaled_full:
            scaled = {k.__class__.__name__: v / (len(validation_loader.dataset) - 1) for k, v in scaled_crit.items()}
            newD = {k.__class__.__name__: v / (len(validation_loader.dataset) - 1) for k, v in unscaled_crit.items()}
            wandb.log({'epoch': epoch,
                       val_or_test: scaled,
                       "unscaled_" + val_or_test: newD})
        else:
            scaled = {k.__class__.__name__: v / (len(validation_loader.dataset) - 1) for k, v in scaled_crit.items()}
            wandb.log({'epoch': epoch, val_or_test: scaled})
    model.train()
    return list(scaled_crit.values())(0)

python – Is there a Numpy or pyTorch function for this code?

Basically is there a Numpy or PyTorch function that does this:

dims = vp_sa_s.size()
        for i in range(dims(0)):
            for j in range(dims(1)):
                for k in range(dims(2)):
                     #to mimic matlab functionality: vp(mdp_data.sa_s)
                    try:
                        
                        vp_sa_s(i,j,k) = vp(mdp_data('sa_s')(i,j,k))
                        
                    except:
                        print('didnt work with' , mdp_data('sa_s'))

Given that vp_sa_s is size (10,5,5) and each value is a valid index vp i.e in range 0-9. vp is size (10,1) with a bunch of random values.

Matlab do it elegantly and quickly with vp(mdp_data.sa_s) which will form a new (10,5,5) matrix. If all values in mdp_data.sa_s are 1, the result would be a (10,5,5) tensor with each value being the 1st value in vp.

Does a function or method that exists that can achieve this in less than O(N^3) time as the above code is terribly inefficient.

Thanks!

python – Is it only me or people are shifting from Tensorflow to Pytorch?

Surely Tensorflow runs the ML industries and Pytorch is, you can say, still used by researchers and students. But, recently I have started to stumbled upon Pytorch code more often then before. I guess people have started using Pytorch for development also and mastering it could give us, as an ML developer, an edge. But still these are my thoughts. I am seeking answer from experts who surely possess more insights in this.

Mask R-CNN optimizer and learning rate scheduler in Pytorch

In the Mask R-CNN paper the optimizer is described as follows training on MS COCO 2014/2015 dataset for instance segmentation (I believe this is the dataset, correct me if this is wrong)

We train on 8 GPUs (so effective minibatch
size is 16) for 160k iterations, with a learning rate of
0.02 which is decreased by 10 at the 120k iteration. We
use a weight decay of 0.0001 and momentum of 0.9. With
ResNeXt (45), we train with 1 image per GPU and the same
number of iterations, with a starting learning rate of 0.01.

I’m trying to write an optimizer and learning rate scheduler in Pytorch for a similar application, to match this description.

For the optimizer I have:

def get_Mask_RCNN_Optimizer(model, learning_rate=0.02):
    optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate, momentum=0.9, weight_decay=0.0001)
    return optimizer

For the learning rate scheduler I have:

def get_MASK_RCNN_LR_Scheduler(optimizer, step_size):
    scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=step_size, gammma=0.1, verbose=True)
    return scheduler

When the authors say “decreased by 10” do they mean divide by 10? Or do they literally mean subtract by 10, in which case we have a negative learning rate, which seems odd/wrong. Any insights appreciated.

How to use have batch norm not forget batch statistics it just used in Pytorch?

I am in an unusual setting where I should not use running statistics (as that would be considered cheating e.g. meta-learning). However, I often run a forward pass on a set of points (5 in fact) and then I want to evaluate only on 1 point using the previous statistics but batch norm forgets the batch statistics it just uses. I’ve tried to hard code the value it should be but I get strange errors (even when I uncomment things like from the pytorch code itself like checking the dimension size).

How do I hardcode the previous batch statistics so that batch norm works on a new single data point and then reset them for a fresh new next batch?

note: I don’t want to change the batch norm layer type.

Sample code I tried:

def set_tracking_running_stats(model):
    for attr in dir(model):
        if 'bn' in attr:
            target_attr = getattr(model, attr)
            target_attr.track_running_stats = True
            target_attr.running_mean = torch.nn.Parameter(torch.zeros(target_attr.num_features, requires_grad=False))
            target_attr.running_var = torch.nn.Parameter(torch.ones(target_attr.num_features, requires_grad=False))
            target_attr.num_batches_tracked = torch.nn.Parameter(torch.tensor(0, dtype=torch.long), requires_grad=False)
            # target_attr.reset_running_stats()
    return

my most comment errors:

    raise ValueError('expected 2D or 3D input (got {}D input)'
ValueError: expected 2D or 3D input (got 1D input)

and

IndexError: Dimension out of range (expected to be in range of (-1, 0), but got 1)

pytorch forum: https://discuss.pytorch.org/t/how-to-use-have-batch-norm-not-forget-batch-statistics-it-just-used/103437

c++ – Libtorch cannot detect the gpu,but pytorch successfully detects the gpu

Environment´╝Ü

win10 + vs2017;

python 3.7.7 + pytorch 1.4.0;

libtorch 1.7.0 + cuda 10.1

In python

print(torch.cuda.is_available()) //True

In libtorch

std::cout << torch::cuda::device_count() << std::endl;//0
std::cout << torch::cuda::is_available() << std::endl;//0
std::cout << torch::cuda::cudnn_is_available() << std::endl;//0
std::cout << torch::hasCUDA() << std::endl;//0

And In the